[PATCH 0/8] Enable virtio-mem-ccw
To test these you'll need: QEMU: v9.2.0-492-gaa910c20ec Kernel: v6.13-rc1~206^2~46^2~3 or newer. Michal Prívozník (8): qemu: Do NOT autoadd NUMA node for s390 qemu_command: Use qemuBuildVirtioDevProps() to build cmd line for virtio-mem and virtio-pmem qemuxmlconftest: Introduce memory-hotplug-virtio-mem-s390x.xml qemu_caps: Introduce QEMU_CAPS_DEVICE_VIRTIO_MEM_CCW qemu: Validate virtio-mem-ccw qemu: Allow virtio-mem on CCW qemuxmlconftest: Update memory-hotplug-virtio-mem-s390x.xml NEWS: Document virtio-mem-ccw NEWS.rst | 5 ++ src/qemu/qemu_capabilities.c | 2 + src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 28 +++++++-- src/qemu/qemu_domain.c | 6 +- src/qemu/qemu_postparse.c | 1 + src/qemu/qemu_validate.c | 35 ++++++++++- .../caps_10.0.0_s390x.xml | 1 + ...hotplug-virtio-mem-s390x.s390x-latest.args | 39 ++++++++++++ ...-hotplug-virtio-mem-s390x.s390x-latest.xml | 60 +++++++++++++++++++ .../memory-hotplug-virtio-mem-s390x.xml | 57 ++++++++++++++++++ tests/qemuxmlconftest.c | 1 + 12 files changed, 227 insertions(+), 9 deletions(-) create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.args create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.xml create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.xml -- 2.45.2
In some cases, we might automatically add a NUMA node. But this doesn't work for s390 really, because in its commit v2.12.0-rc0~41^2~6 QEMU forbade specifying NUMA nodes for s390. Suppress automatic adding of NUMA node on our side. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_domain.c | 3 ++- src/qemu/qemu_postparse.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c7d7ac26ce..6554b992f0 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -7761,7 +7761,8 @@ qemuDomainDefValidateMemoryHotplug(const virDomainDef *def, return 0; } - if (!ARCH_IS_PPC64(def->os.arch)) { + if (!ARCH_IS_PPC64(def->os.arch) && + !ARCH_IS_S390(def->os.arch)) { /* due to guest support, qemu would silently enable NUMA with one node * once the memory hotplug backend is enabled. To avoid possible * confusion we will enforce user originated numa configuration along diff --git a/src/qemu/qemu_postparse.c b/src/qemu/qemu_postparse.c index 1f9077982a..20ee333e0d 100644 --- a/src/qemu/qemu_postparse.c +++ b/src/qemu/qemu_postparse.c @@ -1805,6 +1805,7 @@ qemuDomainDefNumaAutoAdd(virDomainDef *def, if (!abiUpdate || !virDomainDefHasMemoryHotplug(def) || + qemuDomainIsS390CCW(def) || virDomainNumaGetNodeCount(def->numa) > 0) { return 0; } -- 2.45.2
On 24.01.25 13:21, Michal Privoznik wrote:
In some cases, we might automatically add a NUMA node. But this doesn't work for s390 really, because in its commit v2.12.0-rc0~41^2~6 QEMU forbade specifying NUMA nodes for s390. Suppress automatic adding of NUMA node on our side.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> ---
Reviewed-by: David Hildenbrand <david@redhat.com> -- Cheers, David / dhildenb
Reviewed-by: Boris Fiuczynski <fiuczy@linux.ibm.com> On 1/24/25 13:21, Michal Privoznik wrote:
In some cases, we might automatically add a NUMA node. But this doesn't work for s390 really, because in its commit v2.12.0-rc0~41^2~6 QEMU forbade specifying NUMA nodes for s390. Suppress automatic adding of NUMA node on our side.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_domain.c | 3 ++- src/qemu/qemu_postparse.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c7d7ac26ce..6554b992f0 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -7761,7 +7761,8 @@ qemuDomainDefValidateMemoryHotplug(const virDomainDef *def, return 0; }
- if (!ARCH_IS_PPC64(def->os.arch)) { + if (!ARCH_IS_PPC64(def->os.arch) && + !ARCH_IS_S390(def->os.arch)) { /* due to guest support, qemu would silently enable NUMA with one node * once the memory hotplug backend is enabled. To avoid possible * confusion we will enforce user originated numa configuration along diff --git a/src/qemu/qemu_postparse.c b/src/qemu/qemu_postparse.c index 1f9077982a..20ee333e0d 100644 --- a/src/qemu/qemu_postparse.c +++ b/src/qemu/qemu_postparse.c @@ -1805,6 +1805,7 @@ qemuDomainDefNumaAutoAdd(virDomainDef *def,
if (!abiUpdate || !virDomainDefHasMemoryHotplug(def) || + qemuDomainIsS390CCW(def) || virDomainNumaGetNodeCount(def->numa) > 0) { return 0; }
-- Mit freundlichen Grüßen/Kind regards Boris Fiuczynski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
Both, virtio-mem and virtio-pmem devices follow traditional QEMU naming convention: their suffix determines what bus they live on. For instance, virtio-mem-pci, virtio-mem-ccw, virtio-pmem-pci. We already have a function that constructs device name following this convention: qemuBuildVirtioDevGetConfigDev(). While there's no virtio-pmem-ccw device yet, the function can still be used. Another advantage of using the function is - it'll be easier in future when we want to configure various virtio aspects of memory devices (like ats, iommu_platform, etc.). Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_command.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index 1f28de6194..f10d0ffce5 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -959,6 +959,23 @@ qemuBuildVirtioDevGetConfigDev(const virDomainDeviceDef *device, break; } + case VIR_DOMAIN_DEVICE_MEMORY: + switch (device->data.memory->model) { + case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_PMEM: + *baseName = "virtio-pmem"; + break; + case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_MEM: + *baseName = "virtio-mem"; + break; + case VIR_DOMAIN_MEMORY_MODEL_DIMM: + case VIR_DOMAIN_MEMORY_MODEL_NVDIMM: + case VIR_DOMAIN_MEMORY_MODEL_SGX_EPC: + case VIR_DOMAIN_MEMORY_MODEL_NONE: + case VIR_DOMAIN_MEMORY_MODEL_LAST: + break; + } + break; + case VIR_DOMAIN_DEVICE_LEASE: case VIR_DOMAIN_DEVICE_WATCHDOG: case VIR_DOMAIN_DEVICE_GRAPHICS: @@ -971,7 +988,6 @@ qemuBuildVirtioDevGetConfigDev(const virDomainDeviceDef *device, case VIR_DOMAIN_DEVICE_SHMEM: case VIR_DOMAIN_DEVICE_TPM: case VIR_DOMAIN_DEVICE_PANIC: - case VIR_DOMAIN_DEVICE_MEMORY: case VIR_DOMAIN_DEVICE_IOMMU: case VIR_DOMAIN_DEVICE_AUDIO: case VIR_DOMAIN_DEVICE_PSTORE: @@ -3479,12 +3495,16 @@ qemuBuildMemoryDeviceProps(virQEMUDriverConfig *cfg, break; case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_PMEM: - device = "virtio-pmem-pci"; + /* Deliberately not setting @device. */ + if (!(props = qemuBuildVirtioDevProps(VIR_DOMAIN_DEVICE_MEMORY, mem, priv->qemuCaps))) + return NULL; address = mem->target.virtio_pmem.address; break; case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_MEM: - device = "virtio-mem-pci"; + /* Deliberately not setting @device. */ + if (!(props = qemuBuildVirtioDevProps(VIR_DOMAIN_DEVICE_MEMORY, mem, priv->qemuCaps))) + return NULL; if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_DEVICE_VIRTIO_MEM_PCI_PREALLOC) && qemuBuildMemoryGetPagesize(cfg, def, mem, NULL, NULL, NULL, &prealloc) < 0) @@ -3506,7 +3526,7 @@ qemuBuildMemoryDeviceProps(virQEMUDriverConfig *cfg, } if (virJSONValueObjectAdd(&props, - "s:driver", device, + "S:driver", device, "k:node", mem->targetNode, "P:label-size", labelsize * 1024, "P:block-size", blocksize * 1024, -- 2.45.2
Reviewed-by: Boris Fiuczynski <fiuczy@linux.ibm.com> On 1/24/25 13:21, Michal Privoznik wrote:
Both, virtio-mem and virtio-pmem devices follow traditional QEMU naming convention: their suffix determines what bus they live on. For instance, virtio-mem-pci, virtio-mem-ccw, virtio-pmem-pci. We already have a function that constructs device name following this convention: qemuBuildVirtioDevGetConfigDev().
While there's no virtio-pmem-ccw device yet, the function can still be used.
Another advantage of using the function is - it'll be easier in future when we want to configure various virtio aspects of memory devices (like ats, iommu_platform, etc.).
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_command.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index 1f28de6194..f10d0ffce5 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -959,6 +959,23 @@ qemuBuildVirtioDevGetConfigDev(const virDomainDeviceDef *device, break; }
+ case VIR_DOMAIN_DEVICE_MEMORY: + switch (device->data.memory->model) { + case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_PMEM: + *baseName = "virtio-pmem"; + break; + case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_MEM: + *baseName = "virtio-mem"; + break; + case VIR_DOMAIN_MEMORY_MODEL_DIMM: + case VIR_DOMAIN_MEMORY_MODEL_NVDIMM: + case VIR_DOMAIN_MEMORY_MODEL_SGX_EPC: + case VIR_DOMAIN_MEMORY_MODEL_NONE: + case VIR_DOMAIN_MEMORY_MODEL_LAST: + break; + } + break; + case VIR_DOMAIN_DEVICE_LEASE: case VIR_DOMAIN_DEVICE_WATCHDOG: case VIR_DOMAIN_DEVICE_GRAPHICS: @@ -971,7 +988,6 @@ qemuBuildVirtioDevGetConfigDev(const virDomainDeviceDef *device, case VIR_DOMAIN_DEVICE_SHMEM: case VIR_DOMAIN_DEVICE_TPM: case VIR_DOMAIN_DEVICE_PANIC: - case VIR_DOMAIN_DEVICE_MEMORY: case VIR_DOMAIN_DEVICE_IOMMU: case VIR_DOMAIN_DEVICE_AUDIO: case VIR_DOMAIN_DEVICE_PSTORE: @@ -3479,12 +3495,16 @@ qemuBuildMemoryDeviceProps(virQEMUDriverConfig *cfg, break;
case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_PMEM: - device = "virtio-pmem-pci"; + /* Deliberately not setting @device. */ + if (!(props = qemuBuildVirtioDevProps(VIR_DOMAIN_DEVICE_MEMORY, mem, priv->qemuCaps))) + return NULL; address = mem->target.virtio_pmem.address; break;
case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_MEM: - device = "virtio-mem-pci"; + /* Deliberately not setting @device. */ + if (!(props = qemuBuildVirtioDevProps(VIR_DOMAIN_DEVICE_MEMORY, mem, priv->qemuCaps))) + return NULL;
if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_DEVICE_VIRTIO_MEM_PCI_PREALLOC) && qemuBuildMemoryGetPagesize(cfg, def, mem, NULL, NULL, NULL, &prealloc) < 0) @@ -3506,7 +3526,7 @@ qemuBuildMemoryDeviceProps(virQEMUDriverConfig *cfg, }
if (virJSONValueObjectAdd(&props, - "s:driver", device, + "S:driver", device, "k:node", mem->targetNode, "P:label-size", labelsize * 1024, "P:block-size", blocksize * 1024,
-- Mit freundlichen Grüßen/Kind regards Boris Fiuczynski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
This is an example test for virtio-mem on s390. Currently, both virtio-mem devices are placed onto PCI bus, but this will soon change and the test will be updated. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- ...hotplug-virtio-mem-s390x.s390x-latest.args | 41 +++++++++++ ...-hotplug-virtio-mem-s390x.s390x-latest.xml | 71 +++++++++++++++++++ .../memory-hotplug-virtio-mem-s390x.xml | 59 +++++++++++++++ tests/qemuxmlconftest.c | 1 + 4 files changed, 172 insertions(+) create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.args create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.xml create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.xml diff --git a/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.args b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.args new file mode 100644 index 0000000000..9704d7d5e9 --- /dev/null +++ b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.args @@ -0,0 +1,41 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1 \ +USER=test \ +LOGNAME=test \ +XDG_DATA_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.local/share \ +XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.cache \ +XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.config \ +/usr/bin/qemu-system-s390x \ +-name guest=QEMUGuest1,debug-threads=on \ +-S \ +-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain--1-QEMUGuest1/master-key.aes"}' \ +-machine s390-ccw-virtio,usb=off,dump-guest-core=off \ +-accel kvm \ +-cpu gen16a-base \ +-m size=2095104k,maxmem=1099511627776k \ +-overcommit mem-lock=off \ +-smp 2,sockets=2,cores=1,threads=1 \ +-object '{"qom-type":"memory-backend-ram","id":"ram-node0","size":2145386496}' \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ +-uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=charmonitor,fd=1729,server=on,wait=off \ +-mon chardev=charmonitor,id=monitor,mode=control \ +-rtc base=utc \ +-no-shutdown \ +-boot strict=on \ +-device '{"driver":"zpci","uid":1,"fid":0,"target":"pci.1","id":"zpci1"}' \ +-device '{"driver":"pci-bridge","chassis_nr":1,"id":"pci.1","bus":"pci.0","addr":"0x1"}' \ +-object '{"qom-type":"memory-backend-ram","id":"memvirtiomem0","reserve":false,"size":1073741824}' \ +-device '{"driver":"virtio-mem-pci","node":0,"block-size":2097152,"requested-size":536870912,"memdev":"memvirtiomem0","id":"virtiomem0","bus":"pci.0","addr":"0x2"}' \ +-object '{"qom-type":"memory-backend-file","id":"memvirtiomem1","mem-path":"/dev/hugepages2M/libvirt/qemu/-1-QEMUGuest1","reserve":false,"size":2147483648,"host-nodes":[1,2,3],"policy":"bind"}' \ +-device '{"driver":"virtio-mem-pci","node":0,"block-size":2097152,"requested-size":1073741824,"memdev":"memvirtiomem1","prealloc":true,"memaddr":5637144576,"dynamic-memslots":true,"id":"virtiomem1","bus":"pci.1","addr":"0x1"}' \ +-blockdev '{"driver":"host_device","filename":"/dev/HostVG/QEMUGuest1","node-name":"libvirt-1-storage","read-only":false}' \ +-device '{"driver":"virtio-blk-ccw","devno":"fe.0.0000","drive":"libvirt-1-storage","id":"virtio-disk0","bootindex":1}' \ +-audiodev '{"id":"audio1","driver":"none"}' \ +-device '{"driver":"virtio-balloon-ccw","id":"balloon0","devno":"fe.0.0001"}' \ +-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ +-msg timestamp=on diff --git a/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.xml b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.xml new file mode 100644 index 0000000000..336c6e5aac --- /dev/null +++ b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.xml @@ -0,0 +1,71 @@ +<domain type='kvm'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <maxMemory unit='KiB'>1099511627776</maxMemory> + <memory unit='KiB'>8388608</memory> + <currentMemory unit='KiB'>8388608</currentMemory> + <vcpu placement='static' cpuset='0-1'>2</vcpu> + <os> + <type arch='s390x' machine='s390-ccw-virtio'>hvm</type> + <boot dev='hd'/> + </os> + <cpu mode='custom' match='exact' check='none'> + <model fallback='forbid'>gen16a-base</model> + <numa> + <cell id='0' cpus='0-1' memory='2095104' unit='KiB'/> + </numa> + </cpu> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-s390x</emulator> + <disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='/dev/HostVG/QEMUGuest1'/> + <target dev='hda' bus='virtio'/> + <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/> + </disk> + <controller type='pci' index='0' model='pci-root'/> + <controller type='pci' index='1' model='pci-bridge'> + <model name='pci-bridge'/> + <target chassisNr='1'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'> + <zpci uid='0x0001' fid='0x00000000'/> + </address> + </controller> + <audio id='1' type='none'/> + <memballoon model='virtio'> + <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/> + </memballoon> + <panic model='s390'/> + <memory model='virtio-mem'> + <target> + <size unit='KiB'>1048576</size> + <node>0</node> + <block unit='KiB'>2048</block> + <requested unit='KiB'>524288</requested> + </target> + <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'> + <zpci uid='0x0002' fid='0x00000001'/> + </address> + </memory> + <memory model='virtio-mem'> + <source> + <nodemask>1-3</nodemask> + <pagesize unit='KiB'>2048</pagesize> + </source> + <target dynamicMemslots='yes'> + <size unit='KiB'>2097152</size> + <node>0</node> + <block unit='KiB'>2048</block> + <requested unit='KiB'>1048576</requested> + <address base='0x150000000'/> + </target> + <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'> + <zpci uid='0x0003' fid='0x00000002'/> + </address> + </memory> + </devices> +</domain> diff --git a/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.xml b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.xml new file mode 100644 index 0000000000..747877042a --- /dev/null +++ b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.xml @@ -0,0 +1,59 @@ +<domain type='kvm'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <maxMemory unit='KiB'>1099511627776</maxMemory> + <memory unit='KiB'>8388608</memory> + <currentMemory unit='KiB'>8388608</currentMemory> + <vcpu placement='static' cpuset='0-1'>2</vcpu> + <os> + <type arch='s390x' machine='s390-ccw-virtio'>hvm</type> + <boot dev='hd'/> + </os> + <cpu mode='custom' match='exact' check='none'> + <model fallback='forbid'>gen16a-base</model> + <numa> + <cell id='0' cpus='0-1' memory='2095104' unit='KiB'/> + </numa> + </cpu> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-s390x</emulator> + <disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='/dev/HostVG/QEMUGuest1'/> + <target dev='hda' bus='virtio'/> + <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/> + </disk> + <controller type='pci' index='0' model='pci-root'/> + <audio id='1' type='none'/> + <memballoon model='virtio'> + <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/> + </memballoon> + <memory model='virtio-mem'> + <target> + <size unit='KiB'>1048576</size> + <node>0</node> + <block unit='KiB'>2048</block> + <requested unit='KiB'>524288</requested> + </target> + <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> + </memory> + <memory model='virtio-mem'> + <source> + <nodemask>1-3</nodemask> + <pagesize unit='KiB'>2048</pagesize> + </source> + <target dynamicMemslots='yes'> + <size unit='KiB'>2097152</size> + <node>0</node> + <block unit='KiB'>2048</block> + <requested unit='KiB'>1048576</requested> + <address base='0x150000000'/> + </target> + <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> + </memory> + </devices> +</domain> diff --git a/tests/qemuxmlconftest.c b/tests/qemuxmlconftest.c index 5c95a49116..f41e39e313 100644 --- a/tests/qemuxmlconftest.c +++ b/tests/qemuxmlconftest.c @@ -2713,6 +2713,7 @@ mymain(void) * virDomainMemoryDefCheckConflict() works for NVDIMMs which are special * than other memory devices because of how they handle <labelsize/> */ DO_TEST_CAPS_LATEST("memory-hotplug-nvdimm-overlap"); + DO_TEST_CAPS_ARCH_LATEST("memory-hotplug-virtio-mem-s390x", "s390x"); DO_TEST_CAPS_ARCH_LATEST("machine-aeskeywrap-on-caps", "s390x"); DO_TEST_CAPS_ARCH_LATEST("machine-aeskeywrap-on-cap", "s390x"); -- 2.45.2
On 1/24/25 13:21, Michal Privoznik wrote:
This is an example test for virtio-mem on s390. Currently, both virtio-mem devices are placed onto PCI bus, but this will soon change and the test will be updated.
Signed-off-by: Michal Privoznik<mprivozn@redhat.com> --- ...hotplug-virtio-mem-s390x.s390x-latest.args | 41 +++++++++++ ...-hotplug-virtio-mem-s390x.s390x-latest.xml | 71 +++++++++++++++++++ .../memory-hotplug-virtio-mem-s390x.xml | 59 +++++++++++++++ tests/qemuxmlconftest.c | 1 + 4 files changed, 172 insertions(+) create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.args create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.xml create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.xml
I guess with David's response to patch 7 this should not be possible but currently is because of recorded tests/qemucapabilitiesdata/caps_10.0.0_s390x.replies file of qemu v9.2.0-1203-gd6430c17d7. -- Mit freundlichen Grüßen/Kind regards Boris Fiuczynski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On 1/27/25 13:42, Boris Fiuczynski wrote:
On 1/24/25 13:21, Michal Privoznik wrote:
This is an example test for virtio-mem on s390. Currently, both virtio-mem devices are placed onto PCI bus, but this will soon change and the test will be updated.
Signed-off-by: Michal Privoznik<mprivozn@redhat.com> --- ...hotplug-virtio-mem-s390x.s390x-latest.args | 41 +++++++++++ ...-hotplug-virtio-mem-s390x.s390x-latest.xml | 71 +++++++++++++++++++ .../memory-hotplug-virtio-mem-s390x.xml | 59 +++++++++++++++ tests/qemuxmlconftest.c | 1 + 4 files changed, 172 insertions(+) create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem- s390x.s390x-latest.args create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem- s390x.s390x-latest.xml create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem- s390x.xml
I guess with David's response to patch 7 this should not be possible but currently is because of recorded tests/qemucapabilitiesdata/ caps_10.0.0_s390x.replies file of qemu v9.2.0-1203-gd6430c17d7.
Yeah, trying to start such domain results in QEMU throwing an error: qemu-system-s390x: -device virtio-mem-pci: PCI-attached virtio based memory devices not supported But the point of this patch is to make it more obvious how virtio-mem-ccw is configured (patch 7/8). Michal
On 1/27/25 15:34, Michal Prívozník wrote:
On 1/27/25 13:42, Boris Fiuczynski wrote:
On 1/24/25 13:21, Michal Privoznik wrote:
This is an example test for virtio-mem on s390. Currently, both virtio-mem devices are placed onto PCI bus, but this will soon change and the test will be updated.
Signed-off-by: Michal Privoznik<mprivozn@redhat.com> --- ...hotplug-virtio-mem-s390x.s390x-latest.args | 41 +++++++++++ ...-hotplug-virtio-mem-s390x.s390x-latest.xml | 71 +++++++++++++++++++ .../memory-hotplug-virtio-mem-s390x.xml | 59 +++++++++++++++ tests/qemuxmlconftest.c | 1 + 4 files changed, 172 insertions(+) create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem- s390x.s390x-latest.args create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem- s390x.s390x-latest.xml create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem- s390x.xml
I guess with David's response to patch 7 this should not be possible but currently is because of recorded tests/qemucapabilitiesdata/ caps_10.0.0_s390x.replies file of qemu v9.2.0-1203-gd6430c17d7.
Yeah, trying to start such domain results in QEMU throwing an error:
qemu-system-s390x: -device virtio-mem-pci: PCI-attached virtio based memory devices not supported
But the point of this patch is to make it more obvious how virtio-mem-ccw is configured (patch 7/8).
Michal
Looks like the virtio-mem-pci device test should remain and another test virtio-mem-ccw should be added in patch 7. I suggest to adjust the test name(s). With that Reviewed-by: Boris Fiuczynski <fiuczy@linux.ibm.com> -- Mit freundlichen Grüßen/Kind regards Boris Fiuczynski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On 1/27/25 15:46, Boris Fiuczynski wrote:
On 1/27/25 15:34, Michal Prívozník wrote:
On 1/27/25 13:42, Boris Fiuczynski wrote:
On 1/24/25 13:21, Michal Privoznik wrote:
This is an example test for virtio-mem on s390. Currently, both virtio-mem devices are placed onto PCI bus, but this will soon change and the test will be updated.
Signed-off-by: Michal Privoznik<mprivozn@redhat.com> --- ...hotplug-virtio-mem-s390x.s390x-latest.args | 41 +++++++++++ ...-hotplug-virtio-mem-s390x.s390x-latest.xml | 71 ++++++++++++++ +++++ .../memory-hotplug-virtio-mem-s390x.xml | 59 +++++++++++++++ tests/qemuxmlconftest.c | 1 + 4 files changed, 172 insertions(+) create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem- s390x.s390x-latest.args create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem- s390x.s390x-latest.xml create mode 100644 tests/qemuxmlconfdata/memory-hotplug-virtio-mem- s390x.xml
I guess with David's response to patch 7 this should not be possible but currently is because of recorded tests/qemucapabilitiesdata/ caps_10.0.0_s390x.replies file of qemu v9.2.0-1203-gd6430c17d7.
Yeah, trying to start such domain results in QEMU throwing an error:
qemu-system-s390x: -device virtio-mem-pci: PCI-attached virtio based memory devices not supported
But the point of this patch is to make it more obvious how virtio-mem-ccw is configured (patch 7/8).
Michal
Looks like the virtio-mem-pci device test should remain and another test virtio-mem-ccw should be added in patch 7. I suggest to adjust the test name(s).
Yeah, fair enough. So let's wait a bit and see how David's patches end up. After I'll probably respin with v2 (and introduce two test cases).
With that Reviewed-by: Boris Fiuczynski <fiuczy@linux.ibm.com>
Thanks! Michal
This capability tracks whether QEMU supports virtio-mem-ccw device. Introduced in QEMU commit v9.2.0-492-gaa910c20ec only upcoming release of QEMU supports the device. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + tests/qemucapabilitiesdata/caps_10.0.0_s390x.xml | 1 + 3 files changed, 4 insertions(+) diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index 53aa64a009..b712906998 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -725,6 +725,7 @@ VIR_ENUM_IMPL(virQEMUCaps, /* 470 */ "migrate-incoming.exit-on-error", /* QEMU_CAPS_MIGRATE_INCOMING_EXIT_ON_ERROR */ + "virtio-mem-ccw", /* QEMU_CAPS_DEVICE_VIRTIO_MEM_CCW */ ); @@ -1413,6 +1414,7 @@ struct virQEMUCapsStringFlags virQEMUCapsObjectTypes[] = { { "virtio-sound-device", QEMU_CAPS_DEVICE_VIRTIO_SOUND }, { "sev-snp-guest", QEMU_CAPS_SEV_SNP_GUEST }, { "acpi-erst", QEMU_CAPS_DEVICE_ACPI_ERST }, + { "virtio-mem-ccw", QEMU_CAPS_DEVICE_VIRTIO_MEM_CCW }, }; diff --git a/src/qemu/qemu_capabilities.h b/src/qemu/qemu_capabilities.h index 398749a136..b4fc94f2a4 100644 --- a/src/qemu/qemu_capabilities.h +++ b/src/qemu/qemu_capabilities.h @@ -704,6 +704,7 @@ typedef enum { /* virQEMUCapsFlags grouping marker for syntax-check */ /* 470 */ QEMU_CAPS_MIGRATE_INCOMING_EXIT_ON_ERROR, /* exit-on-error argument of migrate-incoming command */ + QEMU_CAPS_DEVICE_VIRTIO_MEM_CCW, /* -device virtio-mem-ccw */ QEMU_CAPS_LAST /* this must always be the last item */ } virQEMUCapsFlags; diff --git a/tests/qemucapabilitiesdata/caps_10.0.0_s390x.xml b/tests/qemucapabilitiesdata/caps_10.0.0_s390x.xml index 242848d6ae..bccce19bfc 100644 --- a/tests/qemucapabilitiesdata/caps_10.0.0_s390x.xml +++ b/tests/qemucapabilitiesdata/caps_10.0.0_s390x.xml @@ -141,6 +141,7 @@ <flag name='netdev-stream-reconnect-miliseconds'/> <flag name='query-cpu-model-expansion.deprecated-props'/> <flag name='migrate-incoming.exit-on-error'/> + <flag name='virtio-mem-ccw'/> <version>9002050</version> <microcodeVersion>39100285</microcodeVersion> <package>v9.2.0-1203-gd6430c17d7</package> -- 2.45.2
Reviewed-by: Boris Fiuczynski <fiuczy@linux.ibm.com> On 1/24/25 13:21, Michal Privoznik wrote:
This capability tracks whether QEMU supports virtio-mem-ccw device. Introduced in QEMU commit v9.2.0-492-gaa910c20ec only upcoming release of QEMU supports the device.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + tests/qemucapabilitiesdata/caps_10.0.0_s390x.xml | 1 + 3 files changed, 4 insertions(+)
diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index 53aa64a009..b712906998 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -725,6 +725,7 @@ VIR_ENUM_IMPL(virQEMUCaps,
/* 470 */ "migrate-incoming.exit-on-error", /* QEMU_CAPS_MIGRATE_INCOMING_EXIT_ON_ERROR */ + "virtio-mem-ccw", /* QEMU_CAPS_DEVICE_VIRTIO_MEM_CCW */ );
@@ -1413,6 +1414,7 @@ struct virQEMUCapsStringFlags virQEMUCapsObjectTypes[] = { { "virtio-sound-device", QEMU_CAPS_DEVICE_VIRTIO_SOUND }, { "sev-snp-guest", QEMU_CAPS_SEV_SNP_GUEST }, { "acpi-erst", QEMU_CAPS_DEVICE_ACPI_ERST }, + { "virtio-mem-ccw", QEMU_CAPS_DEVICE_VIRTIO_MEM_CCW }, };
diff --git a/src/qemu/qemu_capabilities.h b/src/qemu/qemu_capabilities.h index 398749a136..b4fc94f2a4 100644 --- a/src/qemu/qemu_capabilities.h +++ b/src/qemu/qemu_capabilities.h @@ -704,6 +704,7 @@ typedef enum { /* virQEMUCapsFlags grouping marker for syntax-check */
/* 470 */ QEMU_CAPS_MIGRATE_INCOMING_EXIT_ON_ERROR, /* exit-on-error argument of migrate-incoming command */ + QEMU_CAPS_DEVICE_VIRTIO_MEM_CCW, /* -device virtio-mem-ccw */
QEMU_CAPS_LAST /* this must always be the last item */ } virQEMUCapsFlags; diff --git a/tests/qemucapabilitiesdata/caps_10.0.0_s390x.xml b/tests/qemucapabilitiesdata/caps_10.0.0_s390x.xml index 242848d6ae..bccce19bfc 100644 --- a/tests/qemucapabilitiesdata/caps_10.0.0_s390x.xml +++ b/tests/qemucapabilitiesdata/caps_10.0.0_s390x.xml @@ -141,6 +141,7 @@ <flag name='netdev-stream-reconnect-miliseconds'/> <flag name='query-cpu-model-expansion.deprecated-props'/> <flag name='migrate-incoming.exit-on-error'/> + <flag name='virtio-mem-ccw'/> <version>9002050</version> <microcodeVersion>39100285</microcodeVersion> <package>v9.2.0-1203-gd6430c17d7</package>
-- Mit freundlichen Grüßen/Kind regards Boris Fiuczynski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
There are basically two differences between virtio-mem-ccw and virtio-mem-pci. s390 doesn't allow mixing different page sizes and there's no NUMA support in QEMU. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_validate.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/src/qemu/qemu_validate.c b/src/qemu/qemu_validate.c index 086c66b602..d2d8a85bf2 100644 --- a/src/qemu/qemu_validate.c +++ b/src/qemu/qemu_validate.c @@ -5199,7 +5199,8 @@ qemuValidateDomainDeviceDefHub(virDomainHubDef *hub, static int -qemuValidateDomainDeviceDefMemory(virDomainMemoryDef *mem, +qemuValidateDomainDeviceDefMemory(const virDomainMemoryDef *mem, + const virDomainDef *def, virQEMUCaps *qemuCaps) { virSGXCapability *sgxCaps; @@ -5238,12 +5239,40 @@ qemuValidateDomainDeviceDefMemory(virDomainMemoryDef *mem, break; case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_MEM: - if (!virQEMUCapsGet(qemuCaps, QEMU_CAPS_DEVICE_VIRTIO_MEM_PCI)) { + if ((mem->info.type == VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI && + !virQEMUCapsGet(qemuCaps, QEMU_CAPS_DEVICE_VIRTIO_MEM_PCI)) || + (mem->info.type == VIR_DOMAIN_DEVICE_ADDRESS_TYPE_CCW && + !virQEMUCapsGet(qemuCaps, QEMU_CAPS_DEVICE_VIRTIO_MEM_CCW))) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("virtio-mem isn't supported by this QEMU binary")); return -1; } + if (mem->info.type == VIR_DOMAIN_DEVICE_ADDRESS_TYPE_CCW) { + /* virtio-mem-ccw has a few differences compared to virtio-mem-pci: + * + * 1) corresponding memory-backing-* object can't have a different + * page size than the boot memory (see s390_machine_device_plug() + * in qemu sources). + * 2) Since its commit v2.12.0-rc0~41^2~6 QEMU doesn't allow NUMA + * for s390. + */ + + if (mem->source.virtio_mem.pagesize != 0 && + def->mem.nhugepages && + mem->source.virtio_mem.pagesize != def->mem.hugepages[0].size) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("virtio-mem-ccw can't use different page size than the boot memory")); + return -1; + } + + if (mem->targetNode != 0 && mem->targetNode != -1) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("NUMA nodes are not supported for virtio-mem-ccw")); + return -1; + } + } + if (mem->target.virtio_mem.dynamicMemslots == VIR_TRISTATE_BOOL_YES && !virQEMUCapsGet(qemuCaps, QEMU_CAPS_DEVICE_VIRTIO_MEM_PCI_DYNAMIC_MEMSLOTS)) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", @@ -5430,7 +5459,7 @@ qemuValidateDomainDeviceDef(const virDomainDeviceDef *dev, return qemuValidateDomainDeviceDefSound(dev->data.sound, qemuCaps); case VIR_DOMAIN_DEVICE_MEMORY: - return qemuValidateDomainDeviceDefMemory(dev->data.memory, qemuCaps); + return qemuValidateDomainDeviceDefMemory(dev->data.memory, def, qemuCaps); case VIR_DOMAIN_DEVICE_SHMEM: return qemuValidateDomainDeviceDefShmem(dev->data.shmem, qemuCaps); -- 2.45.2
Reviewed-by: Boris Fiuczynski <fiuczy@linux.ibm.com> On 1/24/25 13:21, Michal Privoznik wrote:
There are basically two differences between virtio-mem-ccw and virtio-mem-pci. s390 doesn't allow mixing different page sizes and there's no NUMA support in QEMU.
Signed-off-by: Michal Privoznik<mprivozn@redhat.com> --- src/qemu/qemu_validate.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-)
-- Mit freundlichen Grüßen/Kind regards Boris Fiuczynski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
After previous commits, we can allow virtio-mem to live on CCW channel. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_domain.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 6554b992f0..56f86140cd 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -7673,9 +7673,10 @@ qemuDomainDefValidateMemoryHotplugDevice(const virDomainMemoryDef *mem, case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_MEM: if (mem->info.type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI && + mem->info.type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_CCW && mem->info.type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_NONE) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, - _("only 'pci' addresses are supported for the %1$s device"), + _("only 'pci' and 'ccw' addresses are supported for the %1$s device"), virDomainMemoryModelTypeToString(mem->model)); return -1; } -- 2.45.2
Reviewed-by: Boris Fiuczynski <fiuczy@linux.ibm.com> On 1/24/25 13:21, Michal Privoznik wrote:
After previous commits, we can allow virtio-mem to live on CCW channel.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_domain.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 6554b992f0..56f86140cd 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -7673,9 +7673,10 @@ qemuDomainDefValidateMemoryHotplugDevice(const virDomainMemoryDef *mem,
case VIR_DOMAIN_MEMORY_MODEL_VIRTIO_MEM: if (mem->info.type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI && + mem->info.type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_CCW && mem->info.type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_NONE) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, - _("only 'pci' addresses are supported for the %1$s device"), + _("only 'pci' and 'ccw' addresses are supported for the %1$s device"), virDomainMemoryModelTypeToString(mem->model)); return -1; }
-- Mit freundlichen Grüßen/Kind regards Boris Fiuczynski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
Drop explicit request to place virtio-mem on PCI bus from the input memory-hotplug-virtio-mem-s390x.xml and demonstrate how the device is automatically placed onto CCW. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- ...ory-hotplug-virtio-mem-s390x.s390x-latest.args | 6 ++---- ...mory-hotplug-virtio-mem-s390x.s390x-latest.xml | 15 ++------------- .../memory-hotplug-virtio-mem-s390x.xml | 2 -- 3 files changed, 4 insertions(+), 19 deletions(-) diff --git a/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.args b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.args index 9704d7d5e9..a6bbef5ce7 100644 --- a/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.args +++ b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.args @@ -27,12 +27,10 @@ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.config \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ --device '{"driver":"zpci","uid":1,"fid":0,"target":"pci.1","id":"zpci1"}' \ --device '{"driver":"pci-bridge","chassis_nr":1,"id":"pci.1","bus":"pci.0","addr":"0x1"}' \ -object '{"qom-type":"memory-backend-ram","id":"memvirtiomem0","reserve":false,"size":1073741824}' \ --device '{"driver":"virtio-mem-pci","node":0,"block-size":2097152,"requested-size":536870912,"memdev":"memvirtiomem0","id":"virtiomem0","bus":"pci.0","addr":"0x2"}' \ +-device '{"driver":"virtio-mem-ccw","node":0,"block-size":2097152,"requested-size":536870912,"memdev":"memvirtiomem0","id":"virtiomem0","devno":"fe.0.0002"}' \ -object '{"qom-type":"memory-backend-file","id":"memvirtiomem1","mem-path":"/dev/hugepages2M/libvirt/qemu/-1-QEMUGuest1","reserve":false,"size":2147483648,"host-nodes":[1,2,3],"policy":"bind"}' \ --device '{"driver":"virtio-mem-pci","node":0,"block-size":2097152,"requested-size":1073741824,"memdev":"memvirtiomem1","prealloc":true,"memaddr":5637144576,"dynamic-memslots":true,"id":"virtiomem1","bus":"pci.1","addr":"0x1"}' \ +-device '{"driver":"virtio-mem-ccw","node":0,"block-size":2097152,"requested-size":1073741824,"memdev":"memvirtiomem1","prealloc":true,"memaddr":5637144576,"dynamic-memslots":true,"id":"virtiomem1","devno":"fe.0.0003"}' \ -blockdev '{"driver":"host_device","filename":"/dev/HostVG/QEMUGuest1","node-name":"libvirt-1-storage","read-only":false}' \ -device '{"driver":"virtio-blk-ccw","devno":"fe.0.0000","drive":"libvirt-1-storage","id":"virtio-disk0","bootindex":1}' \ -audiodev '{"id":"audio1","driver":"none"}' \ diff --git a/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.xml b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.xml index 336c6e5aac..fe18b1ec7b 100644 --- a/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.xml +++ b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.s390x-latest.xml @@ -28,13 +28,6 @@ <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/> </disk> <controller type='pci' index='0' model='pci-root'/> - <controller type='pci' index='1' model='pci-bridge'> - <model name='pci-bridge'/> - <target chassisNr='1'/> - <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'> - <zpci uid='0x0001' fid='0x00000000'/> - </address> - </controller> <audio id='1' type='none'/> <memballoon model='virtio'> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/> @@ -47,9 +40,7 @@ <block unit='KiB'>2048</block> <requested unit='KiB'>524288</requested> </target> - <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'> - <zpci uid='0x0002' fid='0x00000001'/> - </address> + <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0002'/> </memory> <memory model='virtio-mem'> <source> @@ -63,9 +54,7 @@ <requested unit='KiB'>1048576</requested> <address base='0x150000000'/> </target> - <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'> - <zpci uid='0x0003' fid='0x00000002'/> - </address> + <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0003'/> </memory> </devices> </domain> diff --git a/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.xml b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.xml index 747877042a..4f9f90d1e2 100644 --- a/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.xml +++ b/tests/qemuxmlconfdata/memory-hotplug-virtio-mem-s390x.xml @@ -39,7 +39,6 @@ <block unit='KiB'>2048</block> <requested unit='KiB'>524288</requested> </target> - <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </memory> <memory model='virtio-mem'> <source> @@ -53,7 +52,6 @@ <requested unit='KiB'>1048576</requested> <address base='0x150000000'/> </target> - <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </memory> </devices> </domain> -- 2.45.2
On 24.01.25 13:21, Michal Privoznik wrote:
Drop explicit request to place virtio-mem on PCI bus from the input memory-hotplug-virtio-mem-s390x.xml and demonstrate how the device is automatically placed onto CCW.
Could it still be manually placed on the PCI bus? As of now, virtio-mem-pci is not supported on s390x -- IIRC plugging the device would fail -- but maybe, in a distant future it might be supported. -- Cheers, David / dhildenb
On 1/24/25 13:35, David Hildenbrand wrote:
On 24.01.25 13:21, Michal Privoznik wrote:
Drop explicit request to place virtio-mem on PCI bus from the input memory-hotplug-virtio-mem-s390x.xml and demonstrate how the device is automatically placed onto CCW.
Could it still be manually placed on the PCI bus?
As of now, virtio-mem-pci is not supported on s390x -- IIRC plugging the device would fail -- but maybe, in a distant future it might be supported.
David, the libvirt probing of capabilities with qemu v9.2.0-1203-gd6430c17d7 returns virtio-mem-pci support based on the QOM. Should that be fixed? -- Mit freundlichen Grüßen/Kind regards Boris Fiuczynski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On 27.01.25 13:01, Boris Fiuczynski wrote:
On 1/24/25 13:35, David Hildenbrand wrote:
On 24.01.25 13:21, Michal Privoznik wrote:
Drop explicit request to place virtio-mem on PCI bus from the input memory-hotplug-virtio-mem-s390x.xml and demonstrate how the device is automatically placed onto CCW.
Could it still be manually placed on the PCI bus?
As of now, virtio-mem-pci is not supported on s390x -- IIRC plugging the device would fail -- but maybe, in a distant future it might be supported.
David,
Hi Boris,
the libvirt probing of capabilities with qemu v9.2.0-1203-gd6430c17d7 returns virtio-mem-pci support based on the QOM. Should that be fixed?
Right, it's similar to virtio-balloon-pci: while it is compiled into QEMU, plugging these devices will fail due to lack of MSI-X support. [1] For virtio-mem-pci, in addition to MSI-X support, we'll have to wire up the (un)plug handlers in the machine, which are currently blocked: Which leaves us with three options: (1) Leave it as is: device is compiled in but cannot be used, just like virtio-balloon-pci (2) Implement MSI-X and (un)plug support (3) Do not compile the device in I did a quick test with (2), but somehow the VM does not detect the device properly. Will do some digging if it can be easily made working. [1] https://lkml.kernel.org/r/20250115161425.246348-1-arbab@linux.ibm.com -- Cheers, David / dhildenb
On 1/27/25 13:44, David Hildenbrand wrote:
On 27.01.25 13:01, Boris Fiuczynski wrote:
On 1/24/25 13:35, David Hildenbrand wrote:
On 24.01.25 13:21, Michal Privoznik wrote:
Drop explicit request to place virtio-mem on PCI bus from the input memory-hotplug-virtio-mem-s390x.xml and demonstrate how the device is automatically placed onto CCW.
Could it still be manually placed on the PCI bus?
It can, but as you point out, qemu refuses to start.
As of now, virtio-mem-pci is not supported on s390x -- IIRC plugging the device would fail -- but maybe, in a distant future it might be supported.
David,
Hi Boris,
the libvirt probing of capabilities with qemu v9.2.0-1203-gd6430c17d7 returns virtio-mem-pci support based on the QOM. Should that be fixed?
Right, it's similar to virtio-balloon-pci: while it is compiled into QEMU, plugging these devices will fail due to lack of MSI-X support. [1]
For virtio-mem-pci, in addition to MSI-X support, we'll have to wire up the (un)plug handlers in the machine, which are currently blocked:
Which leaves us with three options:
(1) Leave it as is: device is compiled in but cannot be used, just like virtio-balloon-pci (2) Implement MSI-X and (un)plug support (3) Do not compile the device in
This last option is something I've experimented with, but then got lost in the weeds of code interdeps. In the end I've decided it's not a huge problem, is it. In my patches (e.g. 5/8) I look what bus is virtio-mem attached to and check for corresponding capability. OTOH, support for .prealloc and .dynamic-memslots is detected from virtio-mem-pci device. If virtio-mem-pci device would be compiled out then libvirt would need a code to check these attributes for virtio-mem-ccw. Michal
On 27.01.25 14:41, Michal Prívozník wrote:
On 1/27/25 13:44, David Hildenbrand wrote:
On 27.01.25 13:01, Boris Fiuczynski wrote:
On 1/24/25 13:35, David Hildenbrand wrote:
On 24.01.25 13:21, Michal Privoznik wrote:
Drop explicit request to place virtio-mem on PCI bus from the input memory-hotplug-virtio-mem-s390x.xml and demonstrate how the device is automatically placed onto CCW.
Could it still be manually placed on the PCI bus?
It can, but as you point out, qemu refuses to start.
As of now, virtio-mem-pci is not supported on s390x -- IIRC plugging the device would fail -- but maybe, in a distant future it might be supported.
David,
Hi Boris,
the libvirt probing of capabilities with qemu v9.2.0-1203-gd6430c17d7 returns virtio-mem-pci support based on the QOM. Should that be fixed?
Right, it's similar to virtio-balloon-pci: while it is compiled into QEMU, plugging these devices will fail due to lack of MSI-X support. [1]
For virtio-mem-pci, in addition to MSI-X support, we'll have to wire up the (un)plug handlers in the machine, which are currently blocked:
Which leaves us with three options:
(1) Leave it as is: device is compiled in but cannot be used, just like virtio-balloon-pci (2) Implement MSI-X and (un)plug support (3) Do not compile the device in
This last option is something I've experimented with, but then got lost in the weeds of code interdeps.
Yes, that last thing happened to me when I last looked into that ...
In the end I've decided it's not a huge problem, is it. In my patches (e.g. 5/8) I look what bus is virtio-mem attached to and check for corresponding capability.
OTOH, support for .prealloc and .dynamic-memslots is detected from virtio-mem-pci device. If virtio-mem-pci device would be compiled out then libvirt would need a code to check these attributes for virtio-mem-ccw.
Yes, that would be unfortunate. Looks like I have it running, so we can just support it in QEMU as it seems. We should default to CCW on s390x, though. -- Cheers, David / dhildenb
On 1/27/25 15:15, David Hildenbrand wrote:
On 27.01.25 14:41, Michal Prívozník wrote:
On 1/27/25 13:44, David Hildenbrand wrote:
On 27.01.25 13:01, Boris Fiuczynski wrote:
On 1/24/25 13:35, David Hildenbrand wrote:
On 24.01.25 13:21, Michal Privoznik wrote:
Drop explicit request to place virtio-mem on PCI bus from the input memory-hotplug-virtio-mem-s390x.xml and demonstrate how the device is automatically placed onto CCW.
Could it still be manually placed on the PCI bus?
It can, but as you point out, qemu refuses to start.
As of now, virtio-mem-pci is not supported on s390x -- IIRC plugging the device would fail -- but maybe, in a distant future it might be supported.
David,
Hi Boris,
the libvirt probing of capabilities with qemu v9.2.0-1203-gd6430c17d7 returns virtio-mem-pci support based on the QOM. Should that be fixed?
Right, it's similar to virtio-balloon-pci: while it is compiled into QEMU, plugging these devices will fail due to lack of MSI-X support. [1]
For virtio-mem-pci, in addition to MSI-X support, we'll have to wire up the (un)plug handlers in the machine, which are currently blocked:
Which leaves us with three options:
(1) Leave it as is: device is compiled in but cannot be used, just like virtio-balloon-pci (2) Implement MSI-X and (un)plug support (3) Do not compile the device in
This last option is something I've experimented with, but then got lost in the weeds of code interdeps.
Yes, that last thing happened to me when I last looked into that ...
In the end I've decided it's not a huge problem, is it. In my patches (e.g. 5/8) I look what bus is virtio-mem attached to and check for corresponding capability.
OTOH, support for .prealloc and .dynamic-memslots is detected from virtio-mem-pci device. If virtio-mem-pci device would be compiled out then libvirt would need a code to check these attributes for virtio- mem-ccw.
Yes, that would be unfortunate.
Looks like I have it running, so we can just support it in QEMU as it seems.
I see you already posted patches. Let me take a look.
We should default to CCW on s390x, though.
Absolutely! And that's what my patches do too. Unless the device is explicitly placed onto PCI bus (<address type='pci'/>) libvirt puts it onto CCW, which in turn selects virtio-mem-ccw when building cmd line. Michal
On 27.01.25 15:40, Michal Prívozník wrote:
On 1/27/25 15:15, David Hildenbrand wrote:
On 27.01.25 14:41, Michal Prívozník wrote:
On 1/27/25 13:44, David Hildenbrand wrote:
On 27.01.25 13:01, Boris Fiuczynski wrote:
On 1/24/25 13:35, David Hildenbrand wrote:
On 24.01.25 13:21, Michal Privoznik wrote: > Drop explicit request to place virtio-mem on PCI bus from the > input memory-hotplug-virtio-mem-s390x.xml and demonstrate how the > device is automatically placed onto CCW. >
Could it still be manually placed on the PCI bus?
It can, but as you point out, qemu refuses to start.
As of now, virtio-mem-pci is not supported on s390x -- IIRC plugging the device would fail -- but maybe, in a distant future it might be supported.
David,
Hi Boris,
the libvirt probing of capabilities with qemu v9.2.0-1203-gd6430c17d7 returns virtio-mem-pci support based on the QOM. Should that be fixed?
Right, it's similar to virtio-balloon-pci: while it is compiled into QEMU, plugging these devices will fail due to lack of MSI-X support. [1]
For virtio-mem-pci, in addition to MSI-X support, we'll have to wire up the (un)plug handlers in the machine, which are currently blocked:
Which leaves us with three options:
(1) Leave it as is: device is compiled in but cannot be used, just like virtio-balloon-pci (2) Implement MSI-X and (un)plug support (3) Do not compile the device in
This last option is something I've experimented with, but then got lost in the weeds of code interdeps.
Yes, that last thing happened to me when I last looked into that ...
In the end I've decided it's not a huge problem, is it. In my patches (e.g. 5/8) I look what bus is virtio-mem attached to and check for corresponding capability.
OTOH, support for .prealloc and .dynamic-memslots is detected from virtio-mem-pci device. If virtio-mem-pci device would be compiled out then libvirt would need a code to check these attributes for virtio- mem-ccw.
Yes, that would be unfortunate.
Looks like I have it running, so we can just support it in QEMU as it seems.
I see you already posted patches. Let me take a look.
We should default to CCW on s390x, though.
Absolutely! And that's what my patches do too. Unless the device is explicitly placed onto PCI bus (<address type='pci'/>) libvirt puts it onto CCW, which in turn selects virtio-mem-ccw when building cmd line.
Great, so it's future-proof to support both. Thanks! -- Cheers, David / dhildenb
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- NEWS.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/NEWS.rst b/NEWS.rst index 5d742df705..3156c18ed0 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -26,6 +26,11 @@ v11.1.0 (unreleased) * **New features** + * Introduce virtio-mem ``<memory/>`` model for s390 guests + + The virtio-mem model of ``<memory/>`` device can now be used with s390 + guests. + * **Improvements** * **Bug fixes** -- 2.45.2
Reviewed-by: Boris Fiuczynski <fiuczy@linux.ibm.com> On 1/24/25 13:21, Michal Privoznik wrote:
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- NEWS.rst | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/NEWS.rst b/NEWS.rst index 5d742df705..3156c18ed0 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -26,6 +26,11 @@ v11.1.0 (unreleased)
* **New features**
+ * Introduce virtio-mem ``<memory/>`` model for s390 guests + + The virtio-mem model of ``<memory/>`` device can now be used with s390 + guests. + * **Improvements**
* **Bug fixes**
-- Mit freundlichen Grüßen/Kind regards Boris Fiuczynski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
participants (4)
-
Boris Fiuczynski -
David Hildenbrand -
Michal Privoznik -
Michal Prívozník