[libvirt] [PATCH 0/2] qemu: Enable NUMA node tag in pci-root for PPC64
This patch addresses the same aspects on PPC the bug 1103314 addressed on x86. It sets the target numa node in the pci-root instead of using the pci-expander-bus. The pci-expander-bus is of not much use on PPC, and the approach is to set the value globally for reasons mentioned in Patch 2 commit message. Known issue : If the destination doesn't recognize the NUMA node in the target, then thats dropped during parsing. There is no ABI failure and the migration will be successful. The issue needs us to relook on how to make sure the unparsed tags on the destination which dont fail the ABI could be enforced during migration. Not addressing that in these two patches. --- Shivaprasad G Bhat (2): qemu: capabilitity: Introduce QEMU_CAPS_SPAPR_PCI_HOST_BRIDGE_NUMA_NODE qemu: Enable NUMA node tag in pci-root for PPC docs/formatdomain.html.in | 5 ++ src/qemu/qemu_capabilities.c | 7 ++ src/qemu/qemu_capabilities.h | 1 src/qemu/qemu_command.c | 25 ++++++++ src/qemu/qemu_domain.c | 15 +++-- .../qemucapabilitiesdata/caps_1.2.2.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_1.3.1.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_1.4.2.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_1.5.3.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_1.6.0.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_1.7.0.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_2.1.1.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_2.4.0.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_2.5.0.x86_64.replies | 22 +++++-- .../caps_2.6.0-gicv2.aarch64.replies | 22 +++++-- .../caps_2.6.0-gicv3.aarch64.replies | 24 +++++--- .../caps_2.6.0.ppc64le.replies | 60 ++++++++++++++++++-- .../qemucapabilitiesdata/caps_2.6.0.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_2.7.0.x86_64.replies | 22 +++++-- ...emuxml2argv-spapr-pci-hos-bridge-numa-node.args | 26 +++++++++ ...qemuxml2argv-spapr-pci-hos-bridge-numa-node.xml | 43 ++++++++++++++ tests/qemuxml2argvtest.c | 2 + 22 files changed, 366 insertions(+), 106 deletions(-) create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-spapr-pci-hos-bridge-numa-node.args create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-spapr-pci-hos-bridge-numa-node.xml -- Signature
The patch adds a capability for spapr-pci-host-bridge.numa_node. Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> --- src/qemu/qemu_capabilities.c | 7 ++ src/qemu/qemu_capabilities.h | 1 .../qemucapabilitiesdata/caps_1.2.2.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_1.3.1.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_1.4.2.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_1.5.3.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_1.6.0.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_1.7.0.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_2.1.1.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_2.4.0.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_2.5.0.x86_64.replies | 22 +++++-- .../caps_2.6.0-gicv2.aarch64.replies | 22 +++++-- .../caps_2.6.0-gicv3.aarch64.replies | 24 +++++--- .../caps_2.6.0.ppc64le.replies | 60 ++++++++++++++++++-- .../qemucapabilitiesdata/caps_2.6.0.x86_64.replies | 22 +++++-- .../qemucapabilitiesdata/caps_2.7.0.x86_64.replies | 22 +++++-- 16 files changed, 257 insertions(+), 99 deletions(-) diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index 747226c..5fe63a3 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -350,6 +350,7 @@ VIR_ENUM_IMPL(virQEMUCaps, QEMU_CAPS_LAST, "ivshmem-plain", "ivshmem-doorbell", /* 240 */ + "spapr-pci-host-bridge.numa_node", ); @@ -1614,6 +1615,10 @@ static struct virQEMUCapsStringFlags virQEMUCapsObjectPropsVirtioNet[] = { { "rx_queue_size", QEMU_CAPS_VIRTIO_NET_RX_QUEUE_SIZE }, }; +static struct virQEMUCapsStringFlags virQEMUCapsObjectPropsSpaprPCIHostBridge[] = { + { "numa_node", QEMU_CAPS_SPAPR_PCI_HOST_BRIDGE_NUMA_NODE }, +}; + static struct virQEMUCapsStringFlags virQEMUCapsObjectPropsVirtioSCSI[] = { { "iothread", QEMU_CAPS_VIRTIO_SCSI_IOTHREAD }, }; @@ -1766,6 +1771,8 @@ static struct virQEMUCapsObjectTypeProps virQEMUCapsObjectProps[] = { ARRAY_CARDINALITY(virQEMUCapsObjectPropsVirtioBalloon) }, { "nec-usb-xhci", virQEMUCapsObjectPropsUSBNECXHCI, ARRAY_CARDINALITY(virQEMUCapsObjectPropsUSBNECXHCI) }, + { "spapr-pci-host-bridge", virQEMUCapsObjectPropsSpaprPCIHostBridge, + ARRAY_CARDINALITY(virQEMUCapsObjectPropsSpaprPCIHostBridge) }, }; struct virQEMUCapsPropTypeObjects { diff --git a/src/qemu/qemu_capabilities.h b/src/qemu/qemu_capabilities.h index d104404..ebab9ea 100644 --- a/src/qemu/qemu_capabilities.h +++ b/src/qemu/qemu_capabilities.h @@ -385,6 +385,7 @@ typedef enum { /* 240 */ QEMU_CAPS_DEVICE_IVSHMEM_DOORBELL, /* -device ivshmem-doorbell */ + QEMU_CAPS_SPAPR_PCI_HOST_BRIDGE_NUMA_NODE, /* spapr-pci-host-bridge.numa_node= */ QEMU_CAPS_LAST /* this must always be the last item */ } virQEMUCapsFlags; diff --git a/tests/qemucapabilitiesdata/caps_1.2.2.x86_64.replies b/tests/qemucapabilitiesdata/caps_1.2.2.x86_64.replies index dff9788..b4f7b9e 100644 --- a/tests/qemucapabilitiesdata/caps_1.2.2.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_1.2.2.x86_64.replies @@ -1701,6 +1701,14 @@ } { + "id": "libvirt-39", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "isapc" @@ -1738,7 +1746,7 @@ "name": "none" } ], - "id": "libvirt-39" + "id": "libvirt-40" } { @@ -1810,7 +1818,7 @@ "name": "Opteron_G4" } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -1818,11 +1826,11 @@ "enabled": false, "present": true }, - "id": "libvirt-41" + "id": "libvirt-42" } { - "id": "libvirt-42", + "id": "libvirt-43", "error": { "class": "CommandNotFound", "desc": "The command query-tpm-models has not been found" @@ -1830,7 +1838,7 @@ } { - "id": "libvirt-43", + "id": "libvirt-44", "error": { "class": "CommandNotFound", "desc": "The command query-tpm-types has not been found" @@ -1838,7 +1846,7 @@ } { - "id": "libvirt-44", + "id": "libvirt-45", "error": { "class": "CommandNotFound", "desc": "The command query-command-line-options has not been found" @@ -1852,5 +1860,5 @@ "capability": "xbzrle" } ], - "id": "libvirt-45" + "id": "libvirt-46" } diff --git a/tests/qemucapabilitiesdata/caps_1.3.1.x86_64.replies b/tests/qemucapabilitiesdata/caps_1.3.1.x86_64.replies index c40f9fe..a5c2310 100644 --- a/tests/qemucapabilitiesdata/caps_1.3.1.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_1.3.1.x86_64.replies @@ -1921,6 +1921,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "q35-next", @@ -1965,7 +1973,7 @@ "name": "none" } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -2043,7 +2051,7 @@ "name": "Opteron_G5" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -2051,11 +2059,11 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { - "id": "libvirt-43", + "id": "libvirt-44", "error": { "class": "CommandNotFound", "desc": "The command query-tpm-models has not been found" @@ -2063,7 +2071,7 @@ } { - "id": "libvirt-44", + "id": "libvirt-45", "error": { "class": "CommandNotFound", "desc": "The command query-tpm-types has not been found" @@ -2071,7 +2079,7 @@ } { - "id": "libvirt-45", + "id": "libvirt-46", "error": { "class": "CommandNotFound", "desc": "The command query-command-line-options has not been found" @@ -2085,5 +2093,5 @@ "capability": "xbzrle" } ], - "id": "libvirt-45" + "id": "libvirt-47" } diff --git a/tests/qemucapabilitiesdata/caps_1.4.2.x86_64.replies b/tests/qemucapabilitiesdata/caps_1.4.2.x86_64.replies index 7e86cff..20c4051 100644 --- a/tests/qemucapabilitiesdata/caps_1.4.2.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_1.4.2.x86_64.replies @@ -1968,6 +1968,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "pc-q35-1.4", @@ -2015,7 +2023,7 @@ "name": "none" } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -2093,7 +2101,7 @@ "name": "qemu64" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -2101,11 +2109,11 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { - "id": "libvirt-43", + "id": "libvirt-44", "error": { "class": "CommandNotFound", "desc": "The command query-tpm-models has not been found" @@ -2113,7 +2121,7 @@ } { - "id": "libvirt-44", + "id": "libvirt-45", "error": { "class": "CommandNotFound", "desc": "The command query-tpm-types has not been found" @@ -2121,7 +2129,7 @@ } { - "id": "libvirt-45", + "id": "libvirt-46", "error": { "class": "CommandNotFound", "desc": "The command query-command-line-options has not been found" @@ -2135,5 +2143,5 @@ "capability": "xbzrle" } ], - "id": "libvirt-46" + "id": "libvirt-47" } diff --git a/tests/qemucapabilitiesdata/caps_1.5.3.x86_64.replies b/tests/qemucapabilitiesdata/caps_1.5.3.x86_64.replies index 7486579..b7a7c04 100644 --- a/tests/qemucapabilitiesdata/caps_1.5.3.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_1.5.3.x86_64.replies @@ -2069,6 +2069,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "pc-q35-1.4", @@ -2138,7 +2146,7 @@ "cpu-max": 1 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -2216,7 +2224,7 @@ "name": "qemu64" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -2224,19 +2232,19 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -2937,7 +2945,7 @@ "option": "drive" } ], - "id": "libvirt-45" + "id": "libvirt-46" } { @@ -2947,5 +2955,5 @@ "capability": "xbzrle" } ], - "id": "libvirt-46" + "id": "libvirt-47" } diff --git a/tests/qemucapabilitiesdata/caps_1.6.0.x86_64.replies b/tests/qemucapabilitiesdata/caps_1.6.0.x86_64.replies index aad28ca..8a9e8a2 100644 --- a/tests/qemucapabilitiesdata/caps_1.6.0.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_1.6.0.x86_64.replies @@ -2114,6 +2114,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "pc-q35-1.4", @@ -2191,7 +2199,7 @@ "cpu-max": 1 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -2269,7 +2277,7 @@ "name": "qemu64" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -2277,19 +2285,19 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -2892,7 +2900,7 @@ "option": "drive" } ], - "id": "libvirt-45" + "id": "libvirt-46" } { @@ -2914,5 +2922,5 @@ "capability": "zero-blocks" } ], - "id": "libvirt-46" + "id": "libvirt-47" } diff --git a/tests/qemucapabilitiesdata/caps_1.7.0.x86_64.replies b/tests/qemucapabilitiesdata/caps_1.7.0.x86_64.replies index b44d5f1..a1bba52 100644 --- a/tests/qemucapabilitiesdata/caps_1.7.0.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_1.7.0.x86_64.replies @@ -2131,6 +2131,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "pc-q35-1.4", @@ -2216,7 +2224,7 @@ "cpu-max": 1 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -2294,7 +2302,7 @@ "name": "qemu64" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -2302,19 +2310,19 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -3107,7 +3115,7 @@ "option": "drive" } ], - "id": "libvirt-45" + "id": "libvirt-46" } { @@ -3129,5 +3137,5 @@ "capability": "zero-blocks" } ], - "id": "libvirt-46" + "id": "libvirt-47" } diff --git a/tests/qemucapabilitiesdata/caps_2.1.1.x86_64.replies b/tests/qemucapabilitiesdata/caps_2.1.1.x86_64.replies index a7d8f36..46cfd84 100644 --- a/tests/qemucapabilitiesdata/caps_2.1.1.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_2.1.1.x86_64.replies @@ -2548,6 +2548,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "pc-1.3", @@ -2649,7 +2657,7 @@ "cpu-max": 255 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -2730,7 +2738,7 @@ "name": "qemu64" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -2738,21 +2746,21 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ "tpm-tis" ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ "passthrough" ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -3612,7 +3620,7 @@ "option": "drive" } ], - "id": "libvirt-45" + "id": "libvirt-46" } { @@ -3634,5 +3642,5 @@ "capability": "zero-blocks" } ], - "id": "libvirt-46" + "id": "libvirt-47" } diff --git a/tests/qemucapabilitiesdata/caps_2.4.0.x86_64.replies b/tests/qemucapabilitiesdata/caps_2.4.0.x86_64.replies index c483c48..5a84934 100644 --- a/tests/qemucapabilitiesdata/caps_2.4.0.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_2.4.0.x86_64.replies @@ -3113,6 +3113,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "pc-i440fx-2.4", @@ -3238,7 +3246,7 @@ "cpu-max": 255 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -3328,7 +3336,7 @@ "name": "qemu64" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -3336,21 +3344,21 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ "tpm-tis" ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ "passthrough" ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -4358,7 +4366,7 @@ "option": "drive" } ], - "id": "libvirt-45" + "id": "libvirt-46" } { @@ -4388,5 +4396,5 @@ "capability": "events" } ], - "id": "libvirt-46" + "id": "libvirt-47" } diff --git a/tests/qemucapabilitiesdata/caps_2.5.0.x86_64.replies b/tests/qemucapabilitiesdata/caps_2.5.0.x86_64.replies index 75a00ce..88801a5 100644 --- a/tests/qemucapabilitiesdata/caps_2.5.0.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_2.5.0.x86_64.replies @@ -3276,6 +3276,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "pc-i440fx-2.4", @@ -3409,7 +3417,7 @@ "cpu-max": 255 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -3499,7 +3507,7 @@ "name": "qemu64" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -3507,21 +3515,21 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ "tpm-tis" ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ "passthrough" ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -4566,7 +4574,7 @@ "option": "drive" } ], - "id": "libvirt-45" + "id": "libvirt-46" } { @@ -4600,5 +4608,5 @@ "capability": "x-postcopy-ram" } ], - "id": "libvirt-46" + "id": "libvirt-47" } diff --git a/tests/qemucapabilitiesdata/caps_2.6.0-gicv2.aarch64.replies b/tests/qemucapabilitiesdata/caps_2.6.0-gicv2.aarch64.replies index 35095b5..f046fc9 100644 --- a/tests/qemucapabilitiesdata/caps_2.6.0-gicv2.aarch64.replies +++ b/tests/qemucapabilitiesdata/caps_2.6.0-gicv2.aarch64.replies @@ -3654,6 +3654,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "integratorcp", @@ -3825,7 +3833,7 @@ "cpu-max": 1 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -3921,7 +3929,7 @@ "name": "pxa270" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -3929,19 +3937,19 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -4947,7 +4955,7 @@ "capability": "postcopy-ram" } ], - "id": "libvirt-46" + "id": "libvirt-47" } { @@ -4963,5 +4971,5 @@ "kernel": true } ], - "id": "libvirt-47" + "id": "libvirt-48" } diff --git a/tests/qemucapabilitiesdata/caps_2.6.0-gicv3.aarch64.replies b/tests/qemucapabilitiesdata/caps_2.6.0-gicv3.aarch64.replies index 0009f8f..a1979a2 100644 --- a/tests/qemucapabilitiesdata/caps_2.6.0-gicv3.aarch64.replies +++ b/tests/qemucapabilitiesdata/caps_2.6.0-gicv3.aarch64.replies @@ -3654,6 +3654,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "integratorcp", @@ -3825,7 +3833,7 @@ "cpu-max": 1 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -3921,7 +3929,7 @@ "name": "pxa270" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -3929,19 +3937,19 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -4913,7 +4921,7 @@ "option": "drive" } ], - "id": "libvirt-45" + "id": "libvirt-46" } { @@ -4947,7 +4955,7 @@ "capability": "postcopy-ram" } ], - "id": "libvirt-46" + "id": "libvirt-47" } { @@ -4963,5 +4971,5 @@ "kernel": false } ], - "id": "libvirt-47" + "id": "libvirt-48" } diff --git a/tests/qemucapabilitiesdata/caps_2.6.0.ppc64le.replies b/tests/qemucapabilitiesdata/caps_2.6.0.ppc64le.replies index 1835b6d..901f526 100644 --- a/tests/qemucapabilitiesdata/caps_2.6.0.ppc64le.replies +++ b/tests/qemucapabilitiesdata/caps_2.6.0.ppc64le.replies @@ -3803,6 +3803,52 @@ { "return": [ { + "name": "dynamic-reconfiguration", + "type": "bool" + }, + { + "name": "dma_win_size", + "type": "uint64" + }, + { + "name": "dma_win_addr", + "type": "uint64" + }, + { + "name": "io_win_size", + "type": "uint64" + }, + { + "name": "mem_win_size", + "type": "uint64" + }, + { + "name": "buid", + "type": "uint64" + }, + { + "name": "io_win_addr", + "type": "uint64" + }, + { + "name": "liobn", + "type": "uint32" + }, + { + "name": "mem_win_addr", + "type": "uint64" + }, + { + "name": "index", + "type": "uint32" + } + ], + "id": "libvirt-40" +} + +{ + "return": [ + { "name": "ref405ep", "cpu-max": 1 }, @@ -3869,7 +3915,7 @@ "cpu-max": 255 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -5165,7 +5211,7 @@ "name": "MPC8541E_v11" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -5173,19 +5219,19 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -6152,7 +6198,7 @@ "option": "drive" } ], - "id": "libvirt-45" + "id": "libvirt-46" } { @@ -6186,5 +6232,5 @@ "capability": "postcopy-ram" } ], - "id": "libvirt-46" + "id": "libvirt-47" } diff --git a/tests/qemucapabilitiesdata/caps_2.6.0.x86_64.replies b/tests/qemucapabilitiesdata/caps_2.6.0.x86_64.replies index 5db3f5a..443b2d6 100644 --- a/tests/qemucapabilitiesdata/caps_2.6.0.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_2.6.0.x86_64.replies @@ -3358,6 +3358,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "name": "pc-0.12", @@ -3467,7 +3475,7 @@ "cpu-max": 255 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -3557,7 +3565,7 @@ "name": "qemu64" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -3565,21 +3573,21 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ "tpm-tis" ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ "passthrough" ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -4673,7 +4681,7 @@ "option": "drive" } ], - "id": "libvirt-45" + "id": "libvirt-46" } { @@ -4707,5 +4715,5 @@ "capability": "postcopy-ram" } ], - "id": "libvirt-46" + "id": "libvirt-47" } diff --git a/tests/qemucapabilitiesdata/caps_2.7.0.x86_64.replies b/tests/qemucapabilitiesdata/caps_2.7.0.x86_64.replies index c525f66..c81fa85 100644 --- a/tests/qemucapabilitiesdata/caps_2.7.0.x86_64.replies +++ b/tests/qemucapabilitiesdata/caps_2.7.0.x86_64.replies @@ -3549,6 +3549,14 @@ } { + "id": "libvirt-40", + "error": { + "class": "DeviceNotFound", + "desc": "Device 'spapr-pci-host-bridge' not found" + } +} + +{ "return": [ { "hotpluggable-cpus": true, @@ -3694,7 +3702,7 @@ "cpu-max": 255 } ], - "id": "libvirt-40" + "id": "libvirt-41" } { @@ -3787,7 +3795,7 @@ "name": "qemu64" } ], - "id": "libvirt-41" + "id": "libvirt-42" } { @@ -3795,21 +3803,21 @@ "enabled": false, "present": true }, - "id": "libvirt-42" + "id": "libvirt-43" } { "return": [ "tpm-tis" ], - "id": "libvirt-43" + "id": "libvirt-44" } { "return": [ "passthrough" ], - "id": "libvirt-44" + "id": "libvirt-45" } { @@ -4907,7 +4915,7 @@ "option": "drive" } ], - "id": "libvirt-45" + "id": "libvirt-46" } { @@ -4941,5 +4949,5 @@ "capability": "postcopy-ram" } ], - "id": "libvirt-46" + "id": "libvirt-47" }
This patch addresses the same aspects on PPC the bug 1103314 addressed on x86. PCI expander bus creates multiple primary PCI busses, where each of these busses can be assigned a specific NUMA affinity, which, on x86 is advertised through ACPI on a per-bus basis. For SPAPR, a PHB's NUMA affinities are assigned on a per-PHB basis, and there is no mechanism for advertising NUMA affinities to a guest on a per-bus basis. So, even if qemu-ppc manages to get some sort of multi-bus topology working using PXB, there is no way to expose the affinities of these busses to the guest. It can only be exposed on a per-PHB/per-domain basis. So, enable NUMA node tag in pci-root controller on PPC. Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> --- docs/formatdomain.html.in | 5 ++ src/qemu/qemu_command.c | 25 +++++++++++- src/qemu/qemu_domain.c | 15 ++++--- ...emuxml2argv-spapr-pci-hos-bridge-numa-node.args | 26 ++++++++++++ ...qemuxml2argv-spapr-pci-hos-bridge-numa-node.xml | 43 ++++++++++++++++++++ tests/qemuxml2argvtest.c | 2 + 6 files changed, 109 insertions(+), 7 deletions(-) create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-spapr-pci-hos-bridge-numa-node.args create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-spapr-pci-hos-bridge-numa-node.xml diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index 11b3330..ea45146 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -3492,6 +3492,11 @@ part of the specified NUMA node (it is up to the user of the libvirt API to attach host devices to the correct pci-expander-bus when assigning them to the domain). + On PPC64, the PCI devices can be specified to be part of a NUMA + node using only the pci-root controller with an optional + <code><node></code> subelement within the + <code><target></code> subelement. All the PCI devices of + the guest will be part of the specified NUMA node. </dd> </dl> <p> diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index 9adf0fe..ec794f0 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -3055,6 +3055,25 @@ qemuBuildControllerDevStr(const virDomainDef *domainDef, return NULL; } +static int qemuBuildSPAPRGlobalPCIRootNodeCommandLine(virCommandPtr cmd, + virDomainControllerDefPtr def, + virQEMUCapsPtr qemuCaps) +{ + if (def->opts.pciopts.numaNode != -1) { + if (!virQEMUCapsGet(qemuCaps, QEMU_CAPS_SPAPR_PCI_HOST_BRIDGE_NUMA_NODE)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("the numa_node option with spapr-pci-host-bridge controller " + "is not supported in this QEMU binary")); + return -1; + } + virCommandAddArg(cmd, "-global"); + virCommandAddArgFormat(cmd, "spapr-pci-host-bridge.numa_node=%d", + def->opts.pciopts.numaNode); + } + + return 0; +} + static int qemuBuildControllerDevCommandLine(virCommandPtr cmd, @@ -3107,8 +3126,12 @@ qemuBuildControllerDevCommandLine(virCommandPtr cmd, /* skip pci-root/pcie-root */ if (cont->type == VIR_DOMAIN_CONTROLLER_TYPE_PCI && (cont->model == VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT || - cont->model == VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT)) + cont->model == VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT)) { + if (ARCH_IS_PPC64(def->os.arch)) + if (qemuBuildSPAPRGlobalPCIRootNodeCommandLine(cmd, cont, qemuCaps) < 0) + return -1; continue; + } /* first SATA controller on Q35 machines is implicit */ if (cont->type == VIR_DOMAIN_CONTROLLER_TYPE_SATA && diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 8cba755..b5f89a6 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -3058,12 +3058,14 @@ qemuDomainDeviceDefPostParse(virDomainDeviceDefPtr dev, /* if a PCI expander bus has a NUMA node set, make sure * that NUMA node is configured in the guest <cpu><numa> * array. NUMA cell id's in this array are numbered - * from 0 .. size-1. + * from 0 .. size-1. Or On PPC, if the pci/pcie-root has the + * NUMA node set, do the same. */ - if ((cont->model == VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS || - cont->model == VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS) && - (int) virDomainNumaGetNodeCount(def->numa) - <= cont->opts.pciopts.numaNode) { + if (((cont->model == VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS || + cont->model == VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS) || + (qemuDomainMachineIsPSeries(def) && + cont->model == VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT)) && + (int) virDomainNumaGetNodeCount(def->numa) <= cont->opts.pciopts.numaNode) { virReportError(VIR_ERR_XML_ERROR, _("%s with index %d is " "configured for a NUMA node (%d) " @@ -3814,7 +3816,8 @@ qemuDomainDefFormatBuf(virQEMUDriverPtr driver, } if (pci && pci->idx == 0 && - pci->model == VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT) { + pci->model == VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT && + pci->opts.pciopts.numaNode == -1) { VIR_DEBUG("Removing default pci-root from domain '%s'" " for migration compatibility", def->name); toremove++; diff --git a/tests/qemuxml2argvdata/qemuxml2argv-spapr-pci-hos-bridge-numa-node.args b/tests/qemuxml2argvdata/qemuxml2argv-spapr-pci-hos-bridge-numa-node.args new file mode 100644 index 0000000..7b70cb6 --- /dev/null +++ b/tests/qemuxml2argvdata/qemuxml2argv-spapr-pci-hos-bridge-numa-node.args @@ -0,0 +1,26 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/home/test \ +USER=test \ +LOGNAME=test \ +QEMU_AUDIO_DRV=none \ +/usr/bin/qemu-system-ppc64 \ +-name QEMUGuest1 \ +-S \ +-M pseries \ +-m 2048 \ +-smp 8,sockets=3,cores=1,threads=8 \ +-numa node,nodeid=0,cpus=0-3,mem=1024 \ +-numa node,nodeid=1,cpus=4-7,mem=1024 \ +-uuid 87eedafe-eedc-4336-8130-ed9fe5dc90c8 \ +-nographic \ +-nodefaults \ +-monitor unix:/tmp/lib/domain--1-QEMUGuest1/monitor.sock,server,nowait \ +-no-acpi \ +-boot c \ +-global spapr-pci-host-bridge.numa_node=1 \ +-device spapr-vscsi,id=scsi0,reg=0x2000 \ +-usb \ +-drive file=/dev/HostVG/QEMUGuest1,format=raw,if=none,id=drive-scsi0-0-0-0 \ +-device scsi-disk,bus=scsi0.0,channel=0,scsi-id=0,lun=0,\ +drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 diff --git a/tests/qemuxml2argvdata/qemuxml2argv-spapr-pci-hos-bridge-numa-node.xml b/tests/qemuxml2argvdata/qemuxml2argv-spapr-pci-hos-bridge-numa-node.xml new file mode 100644 index 0000000..4dcd68b --- /dev/null +++ b/tests/qemuxml2argvdata/qemuxml2argv-spapr-pci-hos-bridge-numa-node.xml @@ -0,0 +1,43 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>87eedafe-eedc-4336-8130-ed9fe5dc90c8</uuid> + <memory unit='KiB'>2097152</memory> + <currentMemory unit='MiB'>2048</currentMemory> + <vcpu placement='static'>8</vcpu> + <numatune> + <memory mode='strict' nodeset='1'/> + </numatune> + <cpu> + <topology sockets='3' cores='1' threads='8'/> + <numa> + <cell id='0' cpus='0-3' memory='1048576' unit='KiB'/> + <cell id='1' cpus='4-7' memory='1048576' unit='KiB'/> + </numa> + </cpu> + <os> + <type arch='ppc64' machine='pseries'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-ppc64</emulator> + <disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='/dev/HostVG/QEMUGuest1'/> + <target dev='hda' bus='scsi'/> + <address type='drive' controller='0' bus='0' target='0' unit='0'/> + </disk> + <controller type='usb' index='0'/> + <controller type='scsi' index='0'/> + <controller type='pci' index='0' model='pci-root'> + <target> + <node>1</node> + </target> + </controller> + <memballoon model='none'/> + <panic model='pseries'/> + </devices> +</domain> diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c index d025930..8a5b96a 100644 --- a/tests/qemuxml2argvtest.c +++ b/tests/qemuxml2argvtest.c @@ -2219,6 +2219,8 @@ mymain(void) QEMU_CAPS_DEVICE_DMI_TO_PCI_BRIDGE, QEMU_CAPS_MACHINE_IOMMU); DO_TEST("cpu-hotplug-startup", QEMU_CAPS_QUERY_HOTPLUGGABLE_CPUS); + DO_TEST("spapr-pci-hos-bridge-numa-node", QEMU_CAPS_NUMA, + QEMU_CAPS_SPAPR_PCI_HOST_BRIDGE_NUMA_NODE); qemuTestDriverFree(&driver);
On Tue, Nov 08, 2016 at 05:35:38PM +0530, Shivaprasad G Bhat wrote:
This patch addresses the same aspects on PPC the bug 1103314 addressed on x86.
PCI expander bus creates multiple primary PCI busses, where each of these busses can be assigned a specific NUMA affinity, which, on x86 is advertised through ACPI on a per-bus basis.
For SPAPR, a PHB's NUMA affinities are assigned on a per-PHB basis, and there is no mechanism for advertising NUMA affinities to a guest on a per-bus basis. So, even if qemu-ppc manages to get some sort of multi-bus topology working using PXB, there is no way to expose the affinities of these busses to the guest. It can only be exposed on a per-PHB/per-domain basis.
So, enable NUMA node tag in pci-root controller on PPC.
That doesn't really solve the bug above properly - your solution here only works if all the PCI devices (whether emulated or host-assigned) are to be associated with the same guest NUMA node. This is a pretty restrictive setup and so of limited use IMHO. It seems like to solve this requires that we create multiple PCI Host Bridges ? Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On 11/08/2016 05:59 PM, Daniel P. Berrange wrote:
That doesn't really solve the bug above properly - your solution here only works if all the PCI devices (whether emulated or host-assigned) are to be associated with the same guest NUMA node. This is a pretty restrictive setup and so of limited use IMHO.
Yes. It is restrictive. There can be no 1:1 pinning of host to guest nodes. But, one can have multiple host nodes pinned to 1 guest node which the host bridge is pinned to. This helps only to the extent that, the guest drivers wont find -1 in the sysfs/numa_node. Some drivers are known to crash when they find -1. The patch addresses that part of the problem.
It seems like to solve this requires that we create multiple PCI Host Bridges ?
Yes. That will be the actual fix to have 1:1 pinning. The current approach keeps that in mind and we can have the numa_node attribute set at the indivudual spapr-pci-host-bridge level when we have it. Then, the individual <pci-root id=X> can have different target node as I have in this patch. Thanks, Shivaprasad
Daniel
Hi Daniel, Can we have this patch for now ? Or should it come along with multiple-phb support ? Thanks, Shivaprasad On 11/08/2016 07:40 PM, Shivaprasad G Bhat wrote:
On 11/08/2016 05:59 PM, Daniel P. Berrange wrote:
That doesn't really solve the bug above properly - your solution here only works if all the PCI devices (whether emulated or host-assigned) are to be associated with the same guest NUMA node. This is a pretty restrictive setup and so of limited use IMHO.
Yes. It is restrictive. There can be no 1:1 pinning of host to guest nodes. But, one can have multiple host nodes pinned to 1 guest node which the host bridge is pinned to.
This helps only to the extent that, the guest drivers wont find -1 in the sysfs/numa_node. Some drivers are known to crash when they find -1. The patch addresses that part of the problem.
It seems like to solve this requires that we create multiple PCI Host Bridges ?
Yes. That will be the actual fix to have 1:1 pinning. The current approach keeps that in mind and we can have the numa_node attribute set at the indivudual spapr-pci-host-bridge level when we have it. Then, the individual <pci-root id=X> can have different target node as I have in this patch.
Thanks, Shivaprasad
Daniel
participants (2)
-
Daniel P. Berrange -
Shivaprasad G Bhat