Building vhost-user For OVS Today Using DPDK 2.0.0

So let’s stop for a moment and discuss the build option IVSHMEM. Enamored at the prospect of building the fastest vSwitch available this might be of interest.

The DPDK IVSHMEM library uses fast zero-copy data sharing among virtual machines. This could mean host-to-guest or guest-to-guest. This solution leverages QEUMU’s IVSHMEM mechanism. The use of this mechanism comes at a cost to both security and data integrity. 

From a security perspective, the virtual machine using shared memory needs to be trusted. As the name implies for this fast mechanism the memory is shared. Also, having all the virtual machines sharing this piece of memory opens up the possibility of corrupted data. In addition, there are no hooks in Openstack to take advantage of this fast method of packet processing. 

After weighing all these options carefully and you still want to build the fastest vSwitch, then I will show you how. To opt out just replace x86_64-ivshmem-linuxapp-gcc with x86_64-native-linuxapp-gcc.

To build and install the library which includes IVSHMEM (shared memory):

make config T=x86_64-ivshmem-linuxapp-gcc
make install T=x86_64-ivshmem-linuxapp-gcc

cd dpdk-2.0.0/lib/librte_vhost/eventfd_link/
make
sudo insmod eventfd_link.ko

git clone https://github.com/openvswitch/ovs.git
cd ovs
./boot.sh
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --with-dpdk=../dpdk-2.0.0/x86_64-ivshmem-linuxapp-gcc/ --enable-ssl

iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=8

mount -t hugetlbfs -o pagesize=1G none /dev/hugepages

# for example
RTE_SDK=”${HOME}/dpdk-2.0.0
RTE_TARGET =”x86_64-ivshmem-linuxapp-gcc”

# Install and bind UIO interface eth0
sudo modprobe uio
sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/igb_uio.ko
sudo ${RTE_SDK}/tools/dpdk_nic_bind.py --bind=igb_uio eth0

# Install and bind VFIO interface eth1
# NOTE: VFIO needs to be supported in the kernel and the BIOS.
sudo modprobe vfio-pci
sudo chmod a+x /dev/vfio
sudo chmod 0666 /dev/vfio/*
sudo ${RTE_SDK}/tools/dpdk_nic_bind.py --bind=vfio-pci eth1

/etc/init.d/openvswitch-switch stop
ovsdb-tool convert /etc/openvswitch/conf.db vswitchd/vswitch.ovsschema

cd ${home}/src/${ovs_path}
git checkout 7762f7c39a8f5f115427b598d9e768f9336af466
patch -p1 <../../dpdk-vhost-user-2.patch
patch -p1 <../../ovs-ctl-add-dpdk.patch

killall ovs-vswitchd
ovs-vswitchd --dpdk -c 0x0FF8 -n 4 --socket-mem 1024,0 -- \
unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info \
--mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log \
--pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach –monitor

/usr/lib/systemd/system/openvswitch-nonetwork.service
/usr/share/openvswitch/scripts/ovs-ctl

systemctl status openvswitch.service
systemctl enable openvswitch.service
systemctl start openvswitch.service

wget https://raw.githubusercontent.com/xsited/ssg/master/scripts/build_ovs_dpdk.sh
chmod +x build_ovs_dpdk.sh
./build_ovs_dpdk.sh

cd src/ovs/
make install

apt-get purge openvswitch-switch
apt-get install openvswitch-switch

# Change DPDK_HOME for your system
DPDK_HOME=
modprobe uio
insmod $(DPDK_HOME)/x86_64-ivshmem-linuxapp-gcc/kmod/igb_uio.ko

insmod $(DPDK_HOME)/lib/librte_vhost/eventfd_link/eventfd_link.ko

mount -t hugetlbfs -o pagesize=1G none /dev/hugepages

$(DPDK_HOME)/tools/dpdk_nic_bind.py --bind=igb_uio p514p1
$(DPDK_HOME)/tools/dpdk_nic_bind.py --bind=igb_uio p514p2
$(DPDK_HOME)/tools/dpdk_nic_bind.py --status

killall ovs-vswitchd
ovs-vswitchd --dpdk -c 0x0FF8 -n 4 --socket-mem 1024,0 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor

ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
ovs-vsctl add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk
ovs-vsctl show
f3f6693e-03aa-4d36-ae09-ff4a17689467
Bridge "br0"
    Port "br0"
        Interface "br0"
        type: internal
    Port "dpdk1"
        Interface "dpdk1"
        type: dpdk
    Port "dpdk0"
        Interface "dpdk0"
        type: dpdk
    ovs_version: "2.3.90"

dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x00000000BDFA9618 000120 (v01 INTEL S2600WP
06222004 INTL 20090903)
[ 0.122335] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap
d2078c106f0466 ecap f020de
[ 0.122342] dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap
d2078c106f0466 ecap f020de
[ 0.122470] IOAPIC id 2 under DRHD base 0xfbffe000 IOMMU 0
[ 0.122472] IOAPIC id 0 under DRHD base 0xebffc000 IOMMU 1
[ 0.122473] IOAPIC id 1 under DRHD base 0xebffc000 IOMMU 1

grep -i huge /proc/meminfo
AnonHugePages: 1312768 kB
HugePages_Total: 60
HugePages_Free: 59
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB

tools/dpdk_nic_bind.py --status
Network devices using DPDK-compatible driver 
============================================
0000:05:00.0 'Ethernet Controller 10-Gigabit X540-AT2' drv=igb_uio unused=
0000:05:00.1 'Ethernet Controller 10-Gigabit X540-AT2' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=p785p1 drv=ixgbe unused=igb_uio
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=p785p2 drv=ixgbe unused=igb_uio
0000:08:00.0 'I350 Gigabit Network Connection' if=eth0 drv=igb unused=igb_uio *Active*
0000:08:00.1 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio *Active*

Other network devices
=====================
<none>