Enable SR-IOV for Intel NICs on Proxmox

If you have a problem with throughput from VM to “external device” (for example, to switch, and etc.), and vice versa, especially if you bring >1Gbit. Then you need to passthrough your NIC device to VM. But, you don’t need to passthrough the NIC entirely. You can use SR-IOV (VF) instead.

Long story short, I’ve a Proxmox running VyOS and RouterOS (for distribution riyter). If I’m using VirtIO on ROS, the cpu usage was high and low throughput. So, I decide to searching for alternative, and I found about SR-IOV

BTW, my NIC is Intel X540. My server is Dell R610 (X5650). Not all Intel NIC are supported SR-IOV. According on Intel support, here is list of supported NIC that support SR-IOV :

  
Step for enable SR-IOV :
1. Enable IOMMU on GRUB, located on /etc/default/grub. Add to “GRUB_CMDLINE_LINUX_DEFAULT=”
  • If you have Intel CPU : intel_iommu=on iommu=pt
  • If you have AMD CPU : amd_iommu=on iommu=pt
Don’t forget to save and exit
2. Load vfio module. Edit file on /etc/modules. Don’t forget to save and exit
 vfio
 vfio_iommu_type1
 vfio_pci
 vfio_virqfd
3. Create file named ixgbe.conf, at /etc/modprobe.d. Then, paste this command inside  /etc/modprobe.d/ixgbe.conf. Don’t forget to save and exit
options ixgbe max_vfs=8
blacklist ixgbevf
options vfio_iommu_type1 allow_unsafe_interrupts=1
I’ve set VF’s into 8, for safety reason and my need is not more than 8. You can adjust the max_vfs as much as you need (maximum 16 VF)
Finally, update grub and initramfs. Paste this command : update-grub && update-initramfs -u -k all
And reboot your Proxmox
Then you can passthrough the Virtual Function into VM. From my NIC, the last even number of PCI ID it refers to “Port A” of my NIC. And the odd number it refers to “Port B” of my NIC. Note that the actual port must be plugged to switch (or another device). Maybe it’s vary for every NIC brand (even with same chip), you can test it yourself

Note for RouterOS (and other guest OS) : you must assign the MAC address of the VF (NOT from actual NIC). Or you will face a problem inside RouterOS (it will like a “new NIC” every reboot the ROS or Proxmox).

You can assign to ip command : ip link set dev enp4s0f0 vf Y mac xx:xx:xx:xx:xx:xx, Where : 
  • enp4s0f0 = you can change to your actual NIC that already enabled for SRIOV
  • vf Y = change Y to VF number that you passthrough to VM
  • xx:xx = change to your MAC (or you can random-generated)
Also you can insert it on boot, So every reboot your Proxmox, it will automatically applied. Create blank file on /etc/network/interfaces.d/ and paste the command inside new file. Don’t forget to add “post-up” before insert the command, so it looks like this : post-up ip link set dev enp4s0f0 vf Y mac xx:xx:xx:xx:xx:xx

(You must change with your actual interface, actual VF number, and your own MAC address)

Add source /etc/network/interfaces.d/* into /etc/network/interfaces
That’s it. Now I can bring up to 2Gbit using SR-IOV, instead of VirtIO (OVS DPDK is too hard for setup). Also the CPU usage both from guest and (especially) host now decreased
Proudly powered by WordPress | Theme: Outfit Blog by Crimson Themes.