[PATCH v3 0/3] Add virt machine support for configuring PCI high memory MMIO size
This patch series adds support for configuring the PCI high memory MMIO window size for aarch64 virt machine types using the highmem-mmio-size feature introduced in QEMU v10.0.0 [1]. It allows users to configure the size of the high memory MMIO window above 4GB, which can be required to support PCI passthrough with devices that have a large BARs. The feature is exposed through the existing pcihole64 element associated with the PCIe root controller: <controller type='pci' index='0' model='pcie-root'> <pcihole64 unit='GiB'>512</pcihole64> </controller> This existing schema supports the same semantics for the QEMU PC machine pci-hole64-size parameter and is a natural fit for supporting the highmem-mmio-size feature on the aarch64 virt machine. This series is applied over master and depends on the recently merged patch [2] that added support for QEMU v10.0.0 aarch64 capabilities. For your convenience, this series is also available on Github [3]. [1] https://github.com/qemu/qemu/commit/f10104aeae3a17f181d5bb37b7fd7dad7fe86cba [2] https://github.com/nvmochs/libvirt/commit/cea2ee1d28780808172911e5c586478d4e... [3] git fetch https://github.com/nvmochs/libvirt.git pci_highmem_mmio_size_pcihole64 Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> Changelog: v3 - Patch 2 updates: + Use positive check in qemuBuildGlobalControllerCommandLine + Only allow pcihole64 for PCIe root controllers with arm virt - Added review tags from Daniel P. Berrangé for patch 1 and 3 v2 - Use existing XML scmhema (pcihole64 element) instead of a new one Matthew R. Ochs (3): qemu: Add capability for PCI high memory MMIO size qemu: Add command line support for PCI high memory MMIO size tests: Add pcihole64 test for virt machine src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 14 ++++++++- src/qemu/qemu_validate.c | 4 ++- .../caps_10.0.0_aarch64.xml | 1 + .../pcihole64-virt.aarch64-latest.args | 31 +++++++++++++++++++ .../pcihole64-virt.aarch64-latest.xml | 29 +++++++++++++++++ tests/qemuxmlconfdata/pcihole64-virt.xml | 17 ++++++++++ tests/qemuxmlconftest.c | 1 + 9 files changed, 98 insertions(+), 2 deletions(-) create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.xml -- 2.46.0
Add QEMU capability for PCI high memory MMIO size configuration: - Add QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE capability - Add capability to virt machine properties - Add highmem-mmio-size virt machine property to aarch64 qemu 10.0.0 capabilities This allows detecting support for the highmem-mmio-size virt machine property in QEMU. Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> --- src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml | 1 + 3 files changed, 4 insertions(+) diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index c24584bf75fd..b600dd38b8fc 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -732,6 +732,7 @@ VIR_ENUM_IMPL(virQEMUCaps, /* 475 */ "virtio-scsi.iothread-mapping", /* QEMU_CAPS_VIRTIO_SCSI_IOTHREAD_MAPPING */ + "machine.virt.highmem-mmio-size", /* QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE */ ); @@ -1771,6 +1772,7 @@ static struct virQEMUCapsStringFlags virQEMUCapsMachinePropsVirt[] = { { "iommu", QEMU_CAPS_MACHINE_VIRT_IOMMU }, { "ras", QEMU_CAPS_MACHINE_VIRT_RAS }, { "aia", QEMU_CAPS_MACHINE_VIRT_AIA }, + { "highmem-mmio-size", QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE }, }; static struct virQEMUCapsStringFlags virQEMUCapsMachinePropsGeneric[] = { diff --git a/src/qemu/qemu_capabilities.h b/src/qemu/qemu_capabilities.h index 40aa7acef42e..df33212a348a 100644 --- a/src/qemu/qemu_capabilities.h +++ b/src/qemu/qemu_capabilities.h @@ -713,6 +713,7 @@ typedef enum { /* virQEMUCapsFlags grouping marker for syntax-check */ /* 475 */ QEMU_CAPS_VIRTIO_SCSI_IOTHREAD_MAPPING, /* virtio-scsi supports per-virtqueue iothread mapping */ + QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE, /* -machine virt,highmem-mmio-size=<size> */ QEMU_CAPS_LAST /* this must always be the last item */ } virQEMUCapsFlags; diff --git a/tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml b/tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml index e14e926c290d..7fd50185575a 100644 --- a/tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml +++ b/tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml @@ -160,6 +160,7 @@ <flag name='blockdev-set-active'/> <flag name='shim'/> <flag name='virtio-scsi.iothread-mapping'/> + <flag name='machine.virt.highmem-mmio-size'/> <version>10000000</version> <microcodeVersion>61700285</microcodeVersion> <package>v10.0.0</package> -- 2.46.0
Add support for generating QEMU command line with PCI high memory MMIO size: - Add highmem-mmio-size to machine command line generation using size conveyed through pcihole64 - Add validation for aarch64/virt machine type requirement - Add capability check for QEMU support This enables configuring the PCI high memory MMIO window size for aarch64 virt machine types using the existing pcihole64 element. Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> --- src/qemu/qemu_command.c | 14 +++++++++++++- src/qemu/qemu_validate.c | 4 +++- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index e6d308534f87..d1ebb4cdf9e7 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -6195,7 +6195,8 @@ qemuBuildGlobalControllerCommandLine(virCommand *cmd, for (i = 0; i < def->ncontrollers; i++) { virDomainControllerDef *cont = def->controllers[i]; if (cont->type == VIR_DOMAIN_CONTROLLER_TYPE_PCI && - cont->opts.pciopts.pcihole64) { + cont->opts.pciopts.pcihole64 && + (qemuDomainIsQ35(def) || qemuDomainIsI440FX(def))) { const char *hoststr = NULL; switch (cont->model) { @@ -7112,6 +7113,17 @@ qemuBuildMachineCommandLine(virCommand *cmd, qemuBuildMachineACPI(&buf, def, qemuCaps); + if (qemuDomainIsARMVirt(def)) { + for (i = 0; i < def->ncontrollers; i++) { + virDomainControllerDef *cont = def->controllers[i]; + if (cont->type == VIR_DOMAIN_CONTROLLER_TYPE_PCI && + cont->opts.pciopts.pcihole64) { + virBufferAsprintf(&buf, ",highmem-mmio-size=%lluK", cont->opts.pciopts.pcihole64size); + break; + } + } + } + virCommandAddArgBuffer(cmd, &buf); return 0; diff --git a/src/qemu/qemu_validate.c b/src/qemu/qemu_validate.c index b2c3c9e2f631..87588024ced4 100644 --- a/src/qemu/qemu_validate.c +++ b/src/qemu/qemu_validate.c @@ -4085,7 +4085,9 @@ qemuValidateDomainDeviceDefControllerPCI(const virDomainControllerDef *cont, case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT: if (pciopts->pcihole64 || pciopts->pcihole64size != 0) { - if (!qemuDomainIsQ35(def)) { + if (!qemuDomainIsQ35(def) && + !(qemuDomainIsARMVirt(def) && virQEMUCapsGet(qemuCaps, + QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE))) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Setting the 64-bit PCI hole size is not supported for machine '%1$s'"), def->os.machine); -- 2.46.0
Add a pcihole64 test for the aarch64 virt machine that verifies the value is propagated to the highmem-mmio-size virt machine parameter. Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> --- .../pcihole64-virt.aarch64-latest.args | 31 +++++++++++++++++++ .../pcihole64-virt.aarch64-latest.xml | 29 +++++++++++++++++ tests/qemuxmlconfdata/pcihole64-virt.xml | 17 ++++++++++ tests/qemuxmlconftest.c | 1 + 4 files changed, 78 insertions(+) create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.xml diff --git a/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args b/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args new file mode 100644 index 000000000000..d2df31ed868e --- /dev/null +++ b/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args @@ -0,0 +1,31 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/var/lib/libvirt/qemu/domain--1-foo \ +USER=test \ +LOGNAME=test \ +XDG_DATA_HOME=/var/lib/libvirt/qemu/domain--1-foo/.local/share \ +XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain--1-foo/.cache \ +XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-foo/.config \ +/usr/bin/qemu-system-aarch64 \ +-name guest=foo,debug-threads=on \ +-S \ +-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain--1-foo/master-key.aes"}' \ +-machine virt,usb=off,gic-version=2,dump-guest-core=off,memory-backend=mach-virt.ram,acpi=off,highmem-mmio-size=536870912K \ +-accel tcg \ +-cpu cortex-a15 \ +-m size=1048576k \ +-object '{"qom-type":"memory-backend-ram","id":"mach-virt.ram","size":1073741824}' \ +-overcommit mem-lock=off \ +-smp 1,sockets=1,cores=1,threads=1 \ +-uuid 6ba7b810-9dad-11d1-80b4-00c04fd430c8 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=charmonitor,fd=1729,server=on,wait=off \ +-mon chardev=charmonitor,id=monitor,mode=control \ +-rtc base=utc \ +-no-shutdown \ +-boot strict=on \ +-audiodev '{"id":"audio1","driver":"none"}' \ +-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ +-msg timestamp=on diff --git a/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml b/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml new file mode 100644 index 000000000000..1fb38b429e9f --- /dev/null +++ b/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml @@ -0,0 +1,29 @@ +<domain type='qemu'> + <name>foo</name> + <uuid>6ba7b810-9dad-11d1-80b4-00c04fd430c8</uuid> + <memory unit='KiB'>1048576</memory> + <currentMemory unit='KiB'>1048576</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='aarch64' machine='virt'>hvm</type> + <boot dev='hd'/> + </os> + <features> + <gic version='2'/> + </features> + <cpu mode='custom' match='exact' check='none'> + <model fallback='forbid'>cortex-a15</model> + </cpu> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-aarch64</emulator> + <controller type='pci' index='0' model='pcie-root'> + <pcihole64 unit='KiB'>536870912</pcihole64> + </controller> + <audio id='1' type='none'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxmlconfdata/pcihole64-virt.xml b/tests/qemuxmlconfdata/pcihole64-virt.xml new file mode 100644 index 000000000000..00bb51ab0e9a --- /dev/null +++ b/tests/qemuxmlconfdata/pcihole64-virt.xml @@ -0,0 +1,17 @@ +<domain type='qemu'> + <name>foo</name> + <uuid>6ba7b810-9dad-11d1-80b4-00c04fd430c8</uuid> + <memory unit='KiB'>1048576</memory> + <currentMemory unit='KiB'>1048576</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='aarch64' machine='virt'>hvm</type> + </os> + <devices> + <emulator>/usr/bin/qemu-system-aarch64</emulator> + <controller type='pci' index='0' model='pcie-root'> + <pcihole64 unit='GiB'>512</pcihole64> + </controller> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxmlconftest.c b/tests/qemuxmlconftest.c index 1f31ec810c7a..7fe51ba4f81c 100644 --- a/tests/qemuxmlconftest.c +++ b/tests/qemuxmlconftest.c @@ -2587,6 +2587,7 @@ mymain(void) DO_TEST_CAPS_LATEST("pcihole64"); DO_TEST_CAPS_LATEST("pcihole64-q35"); + DO_TEST_CAPS_ARCH_LATEST("pcihole64-virt", "aarch64"); DO_TEST_CAPS_ARCH_LATEST("arm-vexpressa9-nodevs", "aarch64"); DO_TEST_CAPS_ARCH_LATEST("arm-vexpressa9-basic", "aarch64"); -- 2.46.0
I have pushed this series On Thu, May 15, 2025 at 07:48:07AM -0700, Matthew R. Ochs wrote:
This patch series adds support for configuring the PCI high memory MMIO window size for aarch64 virt machine types using the highmem-mmio-size feature introduced in QEMU v10.0.0 [1]. It allows users to configure the size of the high memory MMIO window above 4GB, which can be required to support PCI passthrough with devices that have a large BARs.
The feature is exposed through the existing pcihole64 element associated with the PCIe root controller:
<controller type='pci' index='0' model='pcie-root'> <pcihole64 unit='GiB'>512</pcihole64> </controller>
This existing schema supports the same semantics for the QEMU PC machine pci-hole64-size parameter and is a natural fit for supporting the highmem-mmio-size feature on the aarch64 virt machine.
This series is applied over master and depends on the recently merged patch [2] that added support for QEMU v10.0.0 aarch64 capabilities.
For your convenience, this series is also available on Github [3].
[1] https://github.com/qemu/qemu/commit/f10104aeae3a17f181d5bb37b7fd7dad7fe86cba [2] https://github.com/nvmochs/libvirt/commit/cea2ee1d28780808172911e5c586478d4e... [3] git fetch https://github.com/nvmochs/libvirt.git pci_highmem_mmio_size_pcihole64
Signed-off-by: Matthew R. Ochs <mochs@nvidia.com>
Changelog: v3 - Patch 2 updates: + Use positive check in qemuBuildGlobalControllerCommandLine + Only allow pcihole64 for PCIe root controllers with arm virt - Added review tags from Daniel P. Berrangé for patch 1 and 3 v2 - Use existing XML scmhema (pcihole64 element) instead of a new one
Matthew R. Ochs (3): qemu: Add capability for PCI high memory MMIO size qemu: Add command line support for PCI high memory MMIO size tests: Add pcihole64 test for virt machine
src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 14 ++++++++- src/qemu/qemu_validate.c | 4 ++- .../caps_10.0.0_aarch64.xml | 1 + .../pcihole64-virt.aarch64-latest.args | 31 +++++++++++++++++++ .../pcihole64-virt.aarch64-latest.xml | 29 +++++++++++++++++ tests/qemuxmlconfdata/pcihole64-virt.xml | 17 ++++++++++ tests/qemuxmlconftest.c | 1 + 9 files changed, 98 insertions(+), 2 deletions(-) create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.xml
-- 2.46.0
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (3)
-
Daniel P. Berrangé -
Matt Ochs -
Matthew R. Ochs