Re: [libvirt] [PATCH v3 15/18] qemu: auto-add pcie-root-port/dmi-to-pci-bridge controllers as needed

Previously libvirt would only add pci-bridge devices automatically when an address was requested for a device that required a legacy PCI slot and none was available. This patch expands that support to dmi-to-pci-bridge (which is needed in order to add a pci-bridge on a machine with a pcie-root), and pcie-root-port (which is needed to add a hotpluggable PCIe device). It does *not* automatically add pcie-switch-upstream-ports or pcie-switch-downstream-ports (and currently there are no plans for that).   Given the existing code to auto-add pci-bridge devices, automatically adding pcie-root-ports is fairly straightforward. The dmi-to-pci-bridge support is a bit tricky though, for a few reasons:   1) Although the only reason to add a dmi-to-pci-bridge is so that     there is a reasonable place to plug in a pci-bridge controller,     most of the time it's not the presence of a pci-bridge *in the     config* that triggers the requirement to add a dmi-to-pci-bridge.     Rather, it is the presence of a legacy-PCI device in the config,     which triggers auto-add of a pci-bridge, which triggers auto-add of     a dmi-to-pci-bridge (this is handled in     virDomainPCIAddressSetGrow() - if there's a request to add a     pci-bridge we'll check if there is a suitable bus to plug it into;     if not, we first add a dmi-to-pci-bridge).   2) Once there is already a single dmi-to-pci-bridge on the system,     there won't be a need for any more, even if it's full, as long as     there is a pci-bridge with an open slot - you can also plug     pci-bridges into existing pci-bridges. So we have to make sure we     don't add a dmi-to-pci-bridge unless there aren't any     dmi-to-pci-bridges *or* any pci-bridges.   3) Although it is strongly discouraged, it is legal for a pci-bridge     to be directly plugged into pcie-root, and we don't want to     auto-add a dmi-to-pci-bridge if there is already a pci-bridge     that's been forced directly into pcie-root. Finally, although I     fail to see the utility of it, it is legal to have something like     this in the xml:         <controller type='pci' model='pcie-root' index='0'/>       <controller type='pci' model='pci-bridge' index='2'/>       and that will lead to an automatically added dmi-to-pci-bridge at     index=1 (to give the unaddressed pci-bridge a proper place to plug     in):         <controller type='pci' model='dmi-to-pci-bridge' index='1'/>       (for example, see the existing test case     "usb-controller-default-q35"). This is handled in     qemuDomainPCIAddressSetCreate() when it's adding in controllers to     fill holes in the indexes.

Although libvirt will now automatically create a dmi-to-pci-bridge when it's needed, the code still remains for now that forces a dmi-to-pci-bridge on all domains with pcie-root (in qemuDomainDefAddDefaultDevices()). That will be removed in the next patch.   For now, the pcie-root-ports are added one to a slot, which is a bit wasteful and means it will fail after 31 total PCIe devices (30 if there are also some PCI devices), but helps keep the changeset down for this patch. A future patch will have 8 pcie-root-ports sharing the functions on a single slot. ---   src/conf/domain_addr.c                             |  88 ++++++++++--   src/qemu/qemu_domain_address.c                     |  91 ++++++++++---   .../qemuxml2argv-q35-pcie-autoadd.args             |  57 ++++++++   .../qemuxml2argv-q35-pcie-autoadd.xml              |  51 +++++++   tests/qemuxml2argvtest.c                           |  23 ++++   .../qemuxml2xmlout-q35-pcie-autoadd.xml            | 147 +++++++++++++++++++++   tests/qemuxml2xmltest.c                            |  23 ++++   7 files changed, 448 insertions(+), 32 deletions(-)   create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-q35-pcie-autoadd.args   create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-q35-pcie-autoadd.xml   create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-q35-pcie-autoadd.xml   diff --git a/src/conf/domain_addr.c b/src/conf/domain_addr.c index 12b5cb2..5f6f6f7 100644 --- a/src/conf/domain_addr.c +++ b/src/conf/domain_addr.c @@ -82,6 +82,30 @@ virDomainPCIControllerModelToConnectType(virDomainControllerModelPCI model)       return 0;   }    + +static int

+virDomainPCIControllerConnectTypeToModel(virDomainPCIConnectFlags flags) +{ +    if (flags & VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT) +        return VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT_PORT; +    if (flags & VIR_PCI_CONNECT_TYPE_PCIE_SWITCH_UPSTREAM_PORT) +        return VIR_DOMAIN_CONTROLLER_MODEL_PCIE_SWITCH_UPSTREAM_PORT; +    if (flags & VIR_PCI_CONNECT_TYPE_PCIE_SWITCH_DOWNSTREAM_PORT) +        return VIR_DOMAIN_CONTROLLER_MODEL_PCIE_SWITCH_DOWNSTREAM_PORT; +    if (flags & VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE) +        return VIR_DOMAIN_CONTROLLER_MODEL_DMI_TO_PCI_BRIDGE; +    if (flags & VIR_PCI_CONNECT_TYPE_PCI_EXPANDER_BUS) +        return VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS; +    if (flags & VIR_PCI_CONNECT_TYPE_PCIE_EXPANDER_BUS) +        return VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS; +    if (flags & VIR_PCI_CONNECT_TYPE_PCI_BRIDGE) +        return VIR_DOMAIN_CONTROLLER_MODEL_PCI_BRIDGE; + +    /* some connect types don't correspond to a controller model */ +    return -1; +} + +   bool   virDomainPCIAddressFlagsCompatible(virPCIDeviceAddressPtr addr,                                      const char *addrStr,

@@ -349,32 +373,72 @@ virDomainPCIAddressSetGrow(virDomainPCIAddressSetPtr addrs,   {       int add;       size_t i; +    int model;

+    bool needDMIToPCIBridge = false;          add = addr->bus - addrs->nbuses + 1; -    i = addrs->nbuses;       if (add <= 0)           return 0;          /* auto-grow only works when we're adding plain PCI devices */

-    if (!(flags & VIR_PCI_CONNECT_TYPE_PCI_DEVICE)) { -        virReportError(VIR_ERR_INTERNAL_ERROR, "%s", -                       _("Cannot automatically add a new PCI bus for a " -                         "device requiring a slot other than standard PCI.")); +    if (flags & VIR_PCI_CONNECT_TYPE_PCI_DEVICE) { +        model = VIR_DOMAIN_CONTROLLER_MODEL_PCI_BRIDGE; + +        /* if there aren't yet any buses that will accept a +         * pci-bridge, and the caller is asking for one, we'll need to +         * add a dmi-to-pci-bridge first. +         */ +        needDMIToPCIBridge = true; +        for (i = 0; i < addrs->nbuses && needDMIToPCIBridge; i++) { +            if (addrs->buses[i].flags & +                VIR_PCI_CONNECT_TYPE_PCI_BRIDGE)

+                needDMIToPCIBridge = false; +        }

+        if (needDMIToPCIBridge && add == 1) { +            /* we need to add at least two buses - one dmi-to-pci, +             * and the other the requested pci-bridge +             */ +            add++; +            addr->bus++; +        } +    } else if (flags & VIR_PCI_CONNECT_TYPE_PCI_BRIDGE) { +        model = VIR_DOMAIN_CONTROLLER_MODEL_DMI_TO_PCI_BRIDGE;

+    } else if (flags & (VIR_PCI_CONNECT_TYPE_PCIE_DEVICE || +                        VIR_PCI_CONNECT_TYPE_PCIE_SWITCH_UPSTREAM_PORT)) {

+        model = VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT_PORT; +    } else { +        int existingContModel = virDomainPCIControllerConnectTypeToModel(flags);

+        if (existingContModel >= 0) { +            virReportError(VIR_ERR_INTERNAL_ERROR, +                           _("a PCI slot is needed to connect a PCI controller " +                             "model='%s', but none is available, and it " +                             "cannot be automatically added"), +                           virDomainControllerModelPCITypeToString(existingContModel)); +        } else { +            virReportError(VIR_ERR_INTERNAL_ERROR, +                           _("Cannot automatically add a new PCI bus for a " +                             "device with connect Flags %.2x"), flags);

+        }           return -1;       }    +    i = addrs->nbuses; +       if (VIR_EXPAND_N(addrs->buses, addrs->nbuses, add) < 0)           return -1;    -    for (; i < addrs->nbuses; i++) { -        /* Any time we auto-add a bus, we will want a multi-slot -         * bus. Currently the only type of bus we will auto-add is a -         * pci-bridge, which is hot-pluggable and provides standard -         * PCI slots. +    if (needDMIToPCIBridge) { +        /* first of the new buses is dmi-to-pci-bridge, the +         * rest are of the requested type            */ -        virDomainPCIAddressBusSetModel(&addrs->buses[i], -                                       VIR_DOMAIN_CONTROLLER_MODEL_PCI_BRIDGE); +        virDomainPCIAddressBusSetModel(&addrs->buses[i++], +                                       VIR_DOMAIN_CONTROLLER_MODEL_DMI_TO_PCI_BRIDGE);       } + +    for (; i < addrs->nbuses; i++) +        virDomainPCIAddressBusSetModel(&addrs->buses[i], model); +       return add;   }    diff --git a/src/qemu/qemu_domain_address.c b/src/qemu/qemu_domain_address.c index 1ff80e3..a9c4c32 100644 --- a/src/qemu/qemu_domain_address.c +++ b/src/qemu/qemu_domain_address.c @@ -797,6 +797,11 @@ qemuDomainPCIAddressSetCreate(virDomainDefPtr def,   {       virDomainPCIAddressSetPtr addrs;       size_t i; +    bool hasPCIeRoot = false; +    int lowestDMIToPCIBridge = nbuses; +    int lowestUnaddressedPCIBridge = nbuses; +    int lowestAddressedPCIBridge = nbuses;

+    virDomainControllerModelPCI defaultModel;          if ((addrs = virDomainPCIAddressSetAlloc(nbuses)) == NULL)           return NULL; @@ -804,38 +809,84 @@ qemuDomainPCIAddressSetCreate(virDomainDefPtr def,       addrs->nbuses = nbuses;       addrs->dryRun = dryRun;    -    /* As a safety measure, set default model='pci-root' for first pci -     * controller and 'pci-bridge' for all subsequent. After setting -     * those defaults, then scan the config and set the actual model -     * for all addrs[idx]->bus that already have a corresponding -     * controller in the config. -     * -     */ -    if (nbuses > 0) -        virDomainPCIAddressBusSetModel(&addrs->buses[0], -                                       VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT); -    for (i = 1; i < nbuses; i++) { -        virDomainPCIAddressBusSetModel(&addrs->buses[i], -                                       VIR_DOMAIN_CONTROLLER_MODEL_PCI_BRIDGE); -    } -       for (i = 0; i < def->ncontrollers; i++) { -        size_t idx = def->controllers[i]->idx; +        virDomainControllerDefPtr cont = def->controllers[i];

+        size_t idx = cont->idx;    -        if (def->controllers[i]->type != VIR_DOMAIN_CONTROLLER_TYPE_PCI) +        if (cont->type != VIR_DOMAIN_CONTROLLER_TYPE_PCI)               continue;              if (idx >= addrs->nbuses) {               virReportError(VIR_ERR_INTERNAL_ERROR,                              _("Inappropriate new pci controller index %zu " -                             "not found in addrs"), idx); +                             "exceeds addrs array length"), idx);

              goto error;           }    -        if (virDomainPCIAddressBusSetModel(&addrs->buses[idx], -                                           def->controllers[i]->model) < 0) +        if (virDomainPCIAddressBusSetModel(&addrs->buses[idx], cont->model) < 0)               goto error; + +        /* we'll use all this info later to determine if we need +         * to add a dmi-to-pci-bridge due to unaddressed pci-bridge controllers +         */ +        if (cont->model == VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT) { +            hasPCIeRoot = true; +        } else if (cont->model == +                   VIR_DOMAIN_CONTROLLER_MODEL_DMI_TO_PCI_BRIDGE) {

+            if (lowestDMIToPCIBridge > idx) +                lowestDMIToPCIBridge = idx; +        } else if (cont->model == VIR_DOMAIN_CONTROLLER_MODEL_PCI_BRIDGE) { +            if (virDeviceInfoPCIAddressWanted(&cont->info)) { +                if (lowestUnaddressedPCIBridge > idx) +                    lowestUnaddressedPCIBridge = idx; +            } else { +                if (lowestAddressedPCIBridge > idx) +                    lowestAddressedPCIBridge = idx; +            }           } +    } + +    if (nbuses > 0 && !addrs->buses[0].model) { +        if (virDomainPCIAddressBusSetModel(&addrs->buses[0], +                                           VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT) < 0) +            goto error; +    }

+    /* Now fill in a reasonable model for all the buses in the set +     * that don't yet have a corresponding controller in the domain +     * config.  In the rare (and actually fairly idiotic, but still +     * allowed for some reason) case that a domain has 1) a pcie-root +     * at index 0, 2) *no* dmi-to-pci-bridge (or pci-bridge that was +     * manually addressed to sit directly on pcie-root), and 3) does +     * have an unaddressed pci-bridge at an index > 1, then we need to +     * add a dmi-to-pci-bridge. +     */ + +    if (!hasPCIeRoot) +        defaultModel = VIR_DOMAIN_CONTROLLER_MODEL_PCI_BRIDGE;

+    else if (lowestUnaddressedPCIBridge < MIN(lowestAddressedPCIBridge, +                                              lowestDMIToPCIBridge)) +        defaultModel = VIR_DOMAIN_CONTROLLER_MODEL_DMI_TO_PCI_BRIDGE;

+    else +        defaultModel = VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT_PORT; + +    for (i = 1; i < addrs->nbuses; i++) { + +        if (addrs->buses[i].model) +            continue; + +        if (virDomainPCIAddressBusSetModel(&addrs->buses[i], defaultModel) < 0) +            goto error; + +        VIR_DEBUG("Auto-adding <controller type='pci' model='%s' index='%zu'/>", +                  virDomainControllerModelPCITypeToString(defaultModel), i); +        /* only add a single dmi-to-pci-bridge, then add pcie-root-port +         * for any other unspecified controller indexes. +         */ +        if (hasPCIeRoot) +            defaultModel = VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT_PORT;

diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c index a7e21fd..cfa522c 100644 --- a/tests/qemuxml2argvtest.c +++ b/tests/qemuxml2argvtest.c @@ -1739,6 +1739,29 @@ mymain(void)               QEMU_CAPS_PCI_MULTIFUNCTION, QEMU_CAPS_ICH9_USB_EHCI1,               QEMU_CAPS_NEC_USB_XHCI,               QEMU_CAPS_DEVICE_VIDEO_PRIMARY); +    /* same as q35-pcie, but all PCI controllers are added automatically */ +    DO_TEST("q35-pcie-autoadd", +            QEMU_CAPS_VIRTIO_PCI_DISABLE_LEGACY, +            QEMU_CAPS_DEVICE_VIRTIO_RNG, +            QEMU_CAPS_OBJECT_RNG_RANDOM, +            QEMU_CAPS_NETDEV, +            QEMU_CAPS_DEVICE_VIRTIO_NET, +            QEMU_CAPS_DEVICE_VIRTIO_GPU, +            QEMU_CAPS_DEVICE_VIRTIO_GPU_VIRGL, +            QEMU_CAPS_VIRTIO_KEYBOARD, +            QEMU_CAPS_VIRTIO_MOUSE, +            QEMU_CAPS_VIRTIO_TABLET, +            QEMU_CAPS_VIRTIO_INPUT_HOST, +            QEMU_CAPS_VIRTIO_SCSI, +            QEMU_CAPS_FSDEV, +            QEMU_CAPS_FSDEV_WRITEOUT, +            QEMU_CAPS_DEVICE_PCI_BRIDGE, +            QEMU_CAPS_DEVICE_DMI_TO_PCI_BRIDGE, +            QEMU_CAPS_DEVICE_IOH3420, +            QEMU_CAPS_ICH9_AHCI, +            QEMU_CAPS_PCI_MULTIFUNCTION, QEMU_CAPS_ICH9_USB_EHCI1,

+            QEMU_CAPS_NEC_USB_XHCI, +            QEMU_CAPS_DEVICE_VIDEO_PRIMARY);       DO_TEST("pcie-root-port",               QEMU_CAPS_DEVICE_PCI_BRIDGE,               QEMU_CAPS_DEVICE_DMI_TO_PCI_BRIDGE, diff --git a/tests/qemuxml2xmloutdata/qemuxml2xmlout-q35-pcie-autoadd.xml b/tests/qemuxml2xmloutdata/qemuxml2xmlout-q35-pcie-autoadd.xml new file mode 100644 index 0000000..b27dbe7 --- /dev/null +++ b/tests/qemuxml2xmloutdata/qemuxml2xmlout-q35-pcie-autoadd.xml @@ -0,0 +1,147 @@ +<domain type='qemu'> +  <name>q35-test</name> +  <uuid>11dbdcdd-4c3b-482b-8903-9bdb8c0a2774</uuid> +  <memory unit='KiB'>2097152</memory> +  <currentMemory unit='KiB'>2097152</currentMemory> +  <vcpu placement='static' cpuset='0-1'>2</vcpu> +  <os> +    <type arch='x86_64' machine='q35'>hvm</type> +    <boot dev='hd'/> +  </os> +  <clock offset='utc'/> +  <on_poweroff>destroy</on_poweroff> +  <on_reboot>restart</on_reboot> +  <on_crash>destroy</on_crash> +  <devices> +    <emulator>/usr/libexec/qemu-kvm</emulator> +    <disk type='block' device='disk'> +      <source dev='/dev/HostVG/QEMUGuest1'/> +      <target dev='vdb' bus='virtio'/> +      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> +    </disk> +    <controller type='virtio-serial' index='0'> +      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> +    </controller> +    <controller type='scsi' index='0' model='virtio-scsi'> +      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> +    </controller> +    <controller type='usb' index='0' model='nec-xhci'> +      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> +    </controller> +    <controller type='sata' index='0'> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> +    </controller> +    <controller type='pci' index='0' model='pcie-root'/> +    <controller type='pci' index='1' model='dmi-to-pci-bridge'> +      <model name='i82801b11-bridge'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> +    </controller>

+    <controller type='pci' index='2' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='2' port='0x10'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> +    </controller> +    <controller type='pci' index='3' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='3' port='0x18'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> +    </controller> +    <controller type='pci' index='4' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='4' port='0x20'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> +    </controller> +    <controller type='pci' index='5' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='5' port='0x28'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> +    </controller> +    <controller type='pci' index='6' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='6' port='0x30'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> +    </controller> +    <controller type='pci' index='7' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='7' port='0x38'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> +    </controller> +    <controller type='pci' index='8' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='8' port='0x40'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> +    </controller> +    <controller type='pci' index='9' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='9' port='0x48'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> +    </controller> +    <controller type='pci' index='10' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='10' port='0x50'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> +    </controller> +    <controller type='pci' index='11' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='11' port='0x58'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> +    </controller> +    <controller type='pci' index='12' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='12' port='0x60'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> +    </controller> +    <controller type='pci' index='13' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='13' port='0x68'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> +    </controller> +    <controller type='pci' index='14' model='pcie-root-port'> +      <model name='ioh3420'/> +      <target chassis='14' port='0x70'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/> +    </controller> +    <filesystem type='mount' accessmode='passthrough'> +      <source dir='/export/to/guest'/> +      <target dir='/import/from/host'/> +      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> +    </filesystem> +    <interface type='user'> +      <mac address='00:11:22:33:44:55'/> +      <model type='virtio'/> +      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> +    </interface> +    <interface type='user'> +      <mac address='00:11:22:33:44:66'/> +      <model type='e1000e'/> +      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> +    </interface> +    <input type='passthrough' bus='virtio'> +      <source evdev='/dev/input/event1234'/> +      <address type='pci' domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> +    </input> +    <input type='mouse' bus='virtio'> +      <address type='pci' domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/> +    </input> +    <input type='keyboard' bus='virtio'> +      <address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/> +    </input> +    <input type='tablet' bus='virtio'> +      <address type='pci' domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/> +    </input> +    <input type='mouse' bus='ps2'/> +    <input type='keyboard' bus='ps2'/> +    <video> +      <model type='virtio' heads='1' primary='yes'/> +      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> +    </video> +    <memballoon model='virtio'> +      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> +    </memballoon> +    <rng model='virtio'> +      <rate bytes='123' period='1234'/> +      <backend model='random'>/dev/urandom</backend> +      <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> +    </rng>