Prerequisite #
- GPU supports GPU passthrough
- Host chipset supports Intel VT-d or AMD-Vi
- SR-IOV (and virtualization if not) is enabled in BIOS
Enable IOMMU #
Append intel_iommu=on to the GRUB_CMDLINE_LINUX line in GRUB config file at /etc/sysconfig/grub
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/vg0-lv_swap rd.lvm.lv=vg0/lv_root rd.lvm.lv=vg0/lv_swap rhgb quiet intel_iommu=on
Disable Nouveau driver #
Append modprobe.blacklist=nouveau rd.driver.blacklist=nouveau to the same GRUB_CMDLINE_LINUX line
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/vg0-lv_swap rd.lvm.lv=vg0/lv_root rd.lvm.lv=vg0/lv_swap rhgb quiet intel_iommu=on modprobe.blacklist=nouveau rd.driver.blacklist=nouveau
Optional, extra step can be
echo "blacklist nouveau" > /etc/modprobe.d/denylist.conf
echo "options nouveau modeset=0" >> /etc/modprobe.d/denylist.conf
Detach GPU from host #
You cannot add the GPU to the virtual machine if the GPU is bound to the host kernel driver, so you must unbind the GPU device from the host before you can add it to the virtual machine. Host drivers often do not support dynamic unbinding of the GPU, so it is recommended to manually exclude the device from binding to the host drivers.
Get the vendor and device ID
# lspci -Dnn | grep -i NVIDIA
0000:84:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
0000:84:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
Of the output above, [10de:2204] and [10de:1aef] are the vendor and device ID we need.
Similarly, append the pci-stub.ids option, with the vendor-id:device-id above as its value, to the GRUB_CMDLINX_LINUX line in /etc/sysconfig/grub
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/vg0-lv_swap rd.lvm.lv=vg0/lv_root rd.lvm.lv=vg0/lv_swap rhgb quiet intel_iommu=on modprobe.blacklist=nouveau rd.driver.blacklist=nouveau pci-stub.ids=10de:11b4,10de:0e0a"
Regenerate all Dracut initramfs images and update the module dependency list
sudo dracut --regenerate-all --force
sudo depmod -a
Rebuild GRUB #
Regenerate the boot loader configuration using grub2-mkconfig to include the modifications above. If UEFI is enabled
grub2-mkconfig -o /etc/grub2-efi.cfg
If UEFI is disabled
grub2-mkconfig -o /etc/grub2.cfg
Reboot.
Confirm that IOMMU is enabled, the host device is added to the list of pci-stub.ids, and Nouveau is blacklisted
# cat /proc/cmdline
BOOT_IMAGE=(hd4,gpt2)/vmlinuz-4.18.0-553.5.1.el8_10.x86_64 root=/dev/mapper/vg0-lv_root ro crashkernel=auto resume=/dev/mapper/vg0-lv_swap rd.lvm.lv=vg0/lv_root rd.lvm.lv=vg0/lv_swap rhgb quiet intel_iommu=on modprobe.blacklist=nouveau rd.driver.blacklist=nouveau pci-stub.ids=10de:11b4,10de:0e0a"
Now you can passthrough the GPU to a VM and install NVIDA driver in the VM to use the GPU