On Mon, May 12, 2025 at 07:33:37PM +0000, Matt Ochs wrote:
On May 12, 2025, at 5:19 AM, Daniel P. Berrangé <berrange@redhat.com> wrote: On Fri, May 09, 2025 at 07:29:04PM +0000, Matt Ochs wrote:
Would it make sense to just use the existing pcihole64 since [I think] it more or less represents the same concept (setting 64bit MMIO window)?
I'm not sure. I've been struggling to reproduce an effect wit hseting the existing -global q35-pcihost.pci-hole64-size=1048576K settings on x86, and also wondering how it interacts with the previously mentioned -fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=262144
Possibly the former only works with SeaBIOS, and the latter only works with EDK2, but I've not figured out how to prove this.
The qemu docs mention opt/ovmf is specifically for OVMF firmware: https://github.com/qemu/qemu/blob/7be29f2f1a3f5b037d27eedbd5df9f441e8c8c16/d...
The pcihole64 setting can be used with OVMF (see below) and with SEABIOS: https://github.com/libvirt/libvirt/blob/master/docs/formatdomain.rst (see pcihole64)
The X-PciMmio64Mb parameter isn't directly supported in libvirt IIUC. The libvirt XML would need to directly pass qemu command line arguments to use it.
I'm wondering what the semantic difference is between setting the pcihole64 property and the X-PciMmio64Mb fwcfg, in the context of OVMF. The fact that both exist, suggests that there is a meaningful difference, which in turn would mean libvirt might need separate XML attributes for each, which in turn influences how we might choose to design the aarch64 solution.
I'm curious if there's a good way to identify the guest memory map impact, as I'm not finding a clear marker in 'dmesg' that correlates ?
We were able to test this by using OVMF without the dynamic mmio window size patch (i.e. a version older than edk2-stable202211) and guest kernel parameters that are not set to allow re-calculating the MMIO window size by deferring guest resource allocations to the guest kernel (i.e. pci=realloc and pci=nocrs aren't set). With this we could reproduce a 4 GPU VM launch with guest BARs not mapped properly due to running out of space/resources. The BAR mapping failures will be clear in dmesg, with no BAR region mappings in /proc/iomem or output of lspci for the GPUs.
From there we added the pcihole64 attribute to the VM's libvirt definition, setting a 2 TB hole size, and the VM booted with guest GPU BARs mapped properly in dmesg + GPU BAR mappings visible in /proc/iomem and lspci output.
Lastly, observed the same behavior by removing the pcihole64 attribute and setting the X-PciMmio64Mb configuration to 2TB.
Or perhaps that would be too messy or x86-centric and it’s better to go with what you proposed (pcimmio64)?
If the 'pcihole64' setting really is setting the MMIO64 window, then it would be preferrable to re-use the existing setting field.
Per the tests above, pcihole64 is setting the MMIO64 window. The only concern I have with using it is that to date, it has been an x86-centric attribute and tied closely with the qemu -global parameter. I don’t think this is a show-stopper, but will require some code changes to allow it to work with the virt machine and connect it up to a different qemu parameter for that machine.
-matt
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|