On Fri, Jun 27, 2025 at 06:39:24PM -0000, mitchell.augustin(a)canonical.com wrote:
Hi,
I'm trying to get a better understanding of how libvirt VMs interact with the default
QEMU setting for pci-hole64-size on q35 hosts, to assess why my libvirt VMs behave
differently from a similarly configured lxd VM. As I understand, both libvirt and lxd are
using qemu q35 VMs under the hood, and both are inheriting their pci-hole64-size from
qemu's default setting (correct me if that's wrong), but in my tests, I'm
getting different behavior from them. I know lxd is probably out of scope from the libvirt
project perspective, so consider this more of a libvirt question w/ some added lxd
context.
All of this is on a DGX B200 host, which contains large (~180GB VRAM) GPUs.
With libvirt/virt-install, I created a q35 virtual machine with CPU host passthrough and 1
or more GPUs passed-through via --host-device. Without additional modifications, this
works as expected, and I can initialize the GPU driver in the VM and run nvidia-smi.
With lxd (which creates a q35 virtual machine with CPU host passthrough by default), I
attached 1 GPU via "lxc config device add passthroughtest gpu gpu pci=1b:00.0".
On that machine, the pci-hole64-size is too small by default, since I see these in my
dmesg:
[ 1.099110] pci 0000:00:01.5: bridge window [mem size 0x6000000000 64bit pref]:
can't assign; no space
[ 1.120274] pci 0000:00:01.5: bridge window [mem size 0x6000000000 64bit pref]:
can't assign; no space
[ 1.183281] pci 0000:06:00.0: BAR 2 [mem size 0x4000000000 64bit pref]: can't
assign; no space
[ 1.186320] pci 0000:06:00.0: BAR 0 [mem size 0x04000000 64bit pref]: can't assign;
no space
[ 1.189340] pci 0000:06:00.0: BAR 4 [mem size 0x02000000 64bit pref]: can't assign;
no space
and I cannot initialize the GPU driver since the BARs weren't mapped correctly.
When I apply a larger hole size to my lxd VM via `lxc config set passthroughtest
raw.qemu=' -global q35-pcihost.pci-hole64-size=8192G'`, I don't see any
"can't assign; no space" messages, and the driver works as expected.
My question about libvirt is - where (if at all) does libvirt interact with qemu's
pci-hole64-size value? If libvirt does not automatically do something functionally similar
to changing the hole size like I need to do above for lxd, and is in fact just using a
qemu default value, is there some other related interaction happening in libvirt that
might explain why my libvirt VMs don't require a manual change to pci-hole64-size,
despite the fact that the relevant parts of the underlying qemu machine should be the
same?
Unless you set it in the XML we don't do anything with it AFAIK. It
might, however be related to something else, like topology. I would
suggest you dump the qemu command line for both use cases and ask on
qemu ML if you want to get into the differences.
Have a nice day,
Martin