User:R7l/Nvidia GPU passthrough with QEMU on Lenovo ThinkPad P53

GPU passthrough is a technology that allows the Linux kernel to directly present an internal PCI GPU to a virtual machine. It allows you to run a virtual machine with Linux or Windows 10+ with near native performance. In most cases this will be used for gaming or anything that requires GPU performance.

It is also known as IOMMU or VFIO and the best way to get a virtual machine running with GPU passthrough is by using QEMU.

In case of the Thinkpad P53 (and maybe other high end Thinkpads) it does work well due to the muxed GPU hardware and the decent IOMMU layout. The external HDMI port is directly wired to the Nvidia GPU while the internal Intel GPU will only output to the laptop display.

Since it is complicated enough already, this guide will only cover a basic setup for Thinkpad P53 series running Gentoo as host on the laptop display and Windows 10 as guest on a monitor connected to the HDMI port. For this reason it will not include instructions for audio passthrough or dynamic switching of the Nvidia GPU between host and guest and assume the Nvidia GPU to be soley for QEMU. Switching the Nvidia GPU between host and guest is known to work. There are several ways to accomplish this with some of them having issues here and there.

Obstacles
There are a number of obstacles in order to successfully pass the Nvidia GPU into a Windows guest system and make it work with the Nvidia drivers. Most of them are related to decisions by Nvidia. While we only expect the powerful Nvidia GPU to work inside the VM, it needs both GPUs to be passed into the VM. But due to Intel GVT-g technology, we will pass a virtualized GPU to it. This Intel GPU will do nothing more then allow the driver to be installed.

The laptop comes with 3 different versions of vBIOS. We extract one version of it from the BIOS update file and another one with the active Nvidia GPU from within the system. The extracted one from BIOS update needs to be patched against OVMF firmware and the other vBIOS needs to exist inside the VM next to a fake battery.

Hardware requirements
To proceed with the installation you will need a dedicated monitor connected to the HDMI port and also a mouse & keyboard connected to USB. Usage of docking stations hasn't been tested.

Preparation
You need to have the external monitor connected during this setup.

BIOS Settings
While optional, it is recommended to update to the latest BIOS version, check the official page.

The following settings are needed.

Extract activated vBIOS from system
For this step you'll need the Nvidia card to be active and running with propretary drivers within your Gentoo system. It needs to display some output while doing so.

Make sure you have the Nvidia card enabled globally as well as the  USE flag:

If you've changed this setting, please update your system:

Make sure the Nvidia card is loaded and the external monitor is used when starting X11:

Restart your system and make sure the external monitor is showing X11 output. If that's ok, you need to get the Bus ID for the Nvidia card:

Dump activated vBIOS to file
You can now dump the rom. Make sure you have the right Bus ID. With the ID shown from the output above, it would work like this:

Extract vBIOS from BIOS update
We will use VBiosFinder to extract the vBIOS that is part of the BIOS update.

Install VBiosFinder
This a one time task. Once you have the vBIOS extracted you won't need the tool again. There is currently no Ebuild for this tool and dependencies in Portage or any known overlay. The tool is based on Ruby and installing it on Gentoo might cause packages (so called bundles) to be installed by 3rd party package manager. The setup on Gentoo will not be covered as it needs quite a bit of dealing with Ruby. Since we deal with virtual machines, it might quicker to just spin up a throwaway VM based on any other distribution and install it there.

Clone their repository from their Github page and follow the instructions on the site to install the tool and it's dependencies. Make sure to place Rom-Parser (you may need to compile it first) and UEFIExtract as binaries into the  subdirectory.

Your 3rdparty directory should look like this:

Download the BIOS update
If you've not downloaded it already, please head over to the official page and download the  from it. Pick the .exe file for Windows 10 which should be between 15 to 20MB.

Place the downloaded file into the VBiosFinder main directory.

Let VBiosFinder extract the vBIOS
For some reason the VBiosFinder seems to need the full path the downloaded file. If everything is correct, you should see output like this:

The output directory should look like this:

The vBIOS we are looking for would be. Place it to the other files:

Patching OVMF firmware with vBIOS
OVMF is an open-source UEFI firmware for QEMU virtual machines.

There is an Ebuild for the OVMF firmware in Portage but it would need extensive editing to get the vBIOS and other patches included. Even more so, since the patches are not compatible to the version in Portage. We will use an independend approach with Docker.

Install Docker
You need to install Docker on your host system or into the throwaway VM you may have installed earlier.

Download and run the patch
Once you have Docker in place, you can clone the required repository to some place and compile it with the patches:

Once that process is finished, place the extracted vBIOS into the  directory:

Run the patch and check if it was working correctly:

Once done, you should find the patched OVMF files in the  directory:

There you have the patched OVMF firmware with the included vBIOS. Place it to the other files:

Create a fake battery
The Nvidia GPU driver will check for an existing battery in your system. Since passing through the real battery is out of the question, we will need to fake the battery into the virtual machine.

Create a file with the following content:

Convert it into a binary file:

Copy the  to the other files:

QEMU
Set the QEMU targets according to your system:

The QEMU package comes with a larger list of possible USE flags. You may want to check those USE flags and set them according to your planned setup. This is a recommended set of flags:

After reviewing and adding any desired USE flags, emerge :

libvirt
As well as QEMU, libvirt comes with a number of Use flags. Please check those flags and set them according to your setup. These are recommended USE flags for libvirt:

After reviewing and adding any desired USE flags, emerge :

User permissions
After emerging, to run virt-manager as a normal user, ensure each user has been added to the  group:

Starting the service
The service needs to be started. It's also a good idea to enabled in order to be around once we restart the system. This can be done with:

With OpenRC:

With Systemd:

Virt-Manager
Virt-Manager allows you to manage virtual machines using a destop UI.

Install :

Check the IOMMU layout
Passing through any PCI device can only be done with all devices at once within the same group. Except for the PCI root device. The group layout on the P53 is well organized and suitable for this task. You can check the layout like this:

The Nvidia GPU with its audio device should look similar on your system. In order to pass through the Nvidia GPU you'll also need to pass through the audio device.

Blacklist Nvidia Modules
To prevent the host from holding on to the Nvidia GPU we need to blacklist the drivers.

Selecting Nvidia GPU / Audio
You need to pass the device IDs of the PCI devices for Nvidia GPU and Audio. Check the PCI list with  and copy the IDs into this file accordingly:

Adding device IDs like this, will block it from the host system.

Grub
The following parameters need to be added to the Grub command line in order to enable VFIO and IOMMU at boot time:

Creating a virtual Intel GVT-g GPU
As explained before, this virtual GVT-g GPU is needed for the Nvidia drivers to install and run properly. It will be seen in guest as a regular Intel GPU.

Get the ID for your Intel GPU with:

is the ID in question. It should be similar on your system.

List supported MDEV types
If you have VFIO with GVT-g working, you should be able to get a list of possible MDEV types:

It's still ok if you only have one or two of them listed. You need to pick one of them while any of them are good for the task. The difference between them is basically the amout of resources you'll have assigned to it. Since we are not using it other then providing it to the Nvidia driver, the least amout of RAM should do it.

Create a UUID
The UUID will be assigned to the device:

We will use this example in the guide. You can use it too or create one of your own.

Device creation
The virtual GPU device can be created using a single command:

If you want the device to be created at boot time, you can do it with a custom script:

Using OpenRC:

Please add your UUID and make sure the script can be executed:

Using Systemd:

There is no local.d when using Systemd. You need to place the above script somewhere on your system (lets say ), make it executable and write a unit file to start it:

Start the script on boot:

Check the result
If you did everything correct, you should see an output like this after a reboot:

Virtual Machine setup
At this point, you should have everything together to get the virtual machine going with your real Nvidia GPU. In order to prepare the virtual machine, you need to create it without installing Windows right away. We'll need to add everything first and install Windows with the Nvidia GPU in place.

Create a new virtual machine
Create a new virtual machine using Virt-Manager. For this guide we will use the name win10. Select your Windows ISO file and click through the menu and add RAM and CPUs as you see fit. Once you reach, you need to make sure you've checked the option to.

Click  to enter the configuration.

Make the following changes in the upcoming configuration window:

Once you've set this, click  but don't install Windows yet. Just let the counter go through and shut down the virtual machine after it.

Add external keyboard and mouse
The virtual machine will not use the internal laptop keyboard / touchpad. So you need to add the connected USB keyboard and mouse with USB passthrough. This is doable from within Virt-Manager.

Open the VM in Virt-Manager again and go to the configuration window. Click  and select. Add mouse and keyboard from the list to the virtual machine.

Replace the OVMF firmware
We've simply added the default OVMF firmware in the initial configuration window. We will replace it with the one we've patched earlier.

Edit the XML file using the  editor:

Leave the editor and it should save the file.

Add virtual Intel GPU
It's time to add the virtual Intel GPU. For this you'll need the UUID of the device, you've assigned to it earlier:

Exit the editor to save the file. If it complains about a existing bus ID, you need to increment the  value.

Add Nvidia GPU
This is the point where you will add the Nvidia GPU into the virtual machine. You'll also add the Nvidia HDMI audio device. You also need to add the extraced vBIOS rom to it. Otherwise it will just return a blank screen.

Add fake battery
Don't miss out to add the fake battery. Otherwise you might run into trouble installing the Nvidia GPU drivers:

This will also require you to update the XML specifications in the very first line:

Finishing touches
Remove the following devices from the list as they will mostly cause trouble in this setup:

Install Windows
Just install Windows with the Virtio drivers from. The drivers for the Intel GPU should be downloaded by itself after a while. You could trigger this manually in the Windows device manager.

Download Nvidia drivers
Check your Nvidia GPU model:

In our example it's a

You need to visit the Nvidia drivers site and download the driver for your GPU from it.

The selection would be (as of September 2021):

External resources

 * https://github.com/coderobe/VBiosFinder - VBiosFinder
 * https://github.com/T-vK/ovmf-with-vbios-patch - ovmf-with-vbios-patch
 * https://wiki.archlinux.org/title/Intel_GVT-g - Arch Wiki - Intel GVT-g
 * https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF - Arch Wiki - PCI passthrough via OVMF
 * https://www.reddit.com/r/VFIO - Reddit - VFIO
 * https://www.reddit.com/r/VFIO/comments/modwuh/experiencespotentially_a_guide_gvt_dgpu - Reddit - GVT + dGPU passthrough on a laptop (Thinkpad P50 - Muxed)
 * https://www.reddit.com/r/VFIO/comments/ma0s7j/how_to_dump_gpu_vbios_on_linux - Reddit - How to dump GPU VBIOS on linux?
 * https://lantian.pub/en/article/modify-computer/laptop-intel-nvidia-optimus-passthrough.lantian - Lan Tian Blog - Intel and NVIDIA GPU Passthrough on a Optimus MUXless Laptop
 * https://github.com/bryansteiner/gpu-passthrough-tutorial - Bryan Steiner - GPU Passthrough Tutorial
 * https://www.heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough - Heiko Sieger - Running Windows 10 on Linux using KVM with VGA Passthrough