[libvirt-users] pcie-expander-bus doesn't support pcie-pci-bridge and pcie-switch-upstream-port
In libvirt, I found pcie-expander-bus controller doesn't support pcie-to-pci-bridge and pcie-switch-upstream-port. Version: libvirt-4.9 # cat /tmp/c.xml ... <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-expander-bus'> <model name='pxb-pcie'/> <target busNr='3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='pci' index='2' model='pcie-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> ... # virsh -k0 -K0 define /tmp/c.xml error: Failed to define domain from /tmp/c.xml error: XML error: The device at PCI address 0000:01:00.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pcie-to-pci-bridge. # cat /tmp/b.xml ... <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-expander-bus'> <model name='pxb-pcie'/> <target busNr='3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='pci' index='2' model='pcie-switch-upstream-port'> <model name='x3130-upstream'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> ... # virsh -k0 -K0 define /tmp/b.xml error: Failed to define domain from /tmp/b.xml error: XML error: The device at PCI address 0000:01:00.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pci-switch-upstream-port. In function virDomainPCIAddressBusSetModel, I find pcie-expander-bus only supports pcie-root-port and dmi-to-pci-bridge 353 case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS: 354 ┆ /* 32 slots, no hotplug, only accepts pcie-root-port or 355 ┆ ┆* dmi-to-pci-bridge 356 ┆ ┆*/ 357 ┆ bus->flags = (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT | 358 ┆ ┆ ┆ ┆ ┆ VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE); 359 ┆ bus->minSlot = 0; 360 ┆ bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST; 361 ┆ break; But it works in qemu: # /usr/libexec/qemu-kvm -machine q35 -m 1024 -device pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x4 -device pcie-pci-bridge,id=pci.250,bus=pci.1,addr=0x0 -device pcie-pci-bridge,id=pci.251,bus=pci.1,addr=0x1 -spice port=5902,addr=0.0.0.0,disable-ticketing /var/lib/libvirt/images/q35.qcow2 So why pcie-expander-bus doesn't these two controllers? For any concerns in libvirt? Thank you~ -- Best regards, ----------------------------------- Han Han Quality Engineer Redhat. Email: hhan@redhat.com Phone: +861065339333
On Wed, 2018-10-17 at 10:50 +0800, Han Han wrote:
In libvirt, I found pcie-expander-bus controller doesn't support pcie-to-pci-bridge and pcie-switch-upstream-port. [...] # virsh -k0 -K0 define /tmp/c.xml
Aside: the -k and -K virsh options are documented as -k | --keepalive-interval=NUM keepalive interval in seconds, 0 for disable -K | --keepalive-count=NUM number of possible missed keepalive messages So -k0 disables keepalive entirely, making -K0 unnecessary :)
error: Failed to define domain from /tmp/c.xml error: XML error: The device at PCI address 0000:01:00.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pcie-to-pci-bridge. [...] In function virDomainPCIAddressBusSetModel, I find pcie-expander-bus only supports pcie-root-port and dmi-to-pci-bridge 353 case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS: 354 ┆ /* 32 slots, no hotplug, only accepts pcie-root-port or 355 ┆ ┆* dmi-to-pci-bridge 356 ┆ ┆*/ 357 ┆ bus->flags = (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT | 358 ┆ ┆ ┆ ┆ ┆ VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE); 359 ┆ bus->minSlot = 0; 360 ┆ bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST; 361 ┆ break;
The above is consistent with pcie-root itself, which also doesn't support plugging a pcie-to-pci-bridge directly into it but requires a pcie-root <- pcie-root-port <- pcie-to-pci-bridge topology. Now, I don't quite recall *why* that is the case - perhaps Laine does? - but I'm sure we had very compelling reasons O:-)
But it works in qemu: # /usr/libexec/qemu-kvm -machine q35 -m 1024 -device pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x4 -device pcie-pci-bridge,id=pci.250,bus=pci.1,addr=0x0 -device pcie-pci-bridge,id=pci.251,bus=pci.1,addr=0x1 -spice port=5902,addr=0.0.0.0,disable-ticketing /var/lib/libvirt/images/q35.qcow2
It should be noted that QEMU in general allows users to make pretty unwise device placement choices without so much of a warning, so I would not take the above as proof the pcie-to-pci-bridge should be allowed to plug into pcie-expander-bus (or pcie-root) directly :) -- Andrea Bolognani / Red Hat / Virtualization
On 10/17/2018 08:56 AM, Andrea Bolognani wrote:
On Wed, 2018-10-17 at 10:50 +0800, Han Han wrote:
In libvirt, I found pcie-expander-bus controller doesn't support pcie-to-pci-bridge and pcie-switch-upstream-port. [...] # virsh -k0 -K0 define /tmp/c.xml Aside: the -k and -K virsh options are documented as
-k | --keepalive-interval=NUM keepalive interval in seconds, 0 for disable -K | --keepalive-count=NUM number of possible missed keepalive messages
So -k0 disables keepalive entirely, making -K0 unnecessary :)
error: Failed to define domain from /tmp/c.xml error: XML error: The device at PCI address 0000:01:00.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pcie-to-pci-bridge. [...] In function virDomainPCIAddressBusSetModel, I find pcie-expander-bus only supports pcie-root-port and dmi-to-pci-bridge 353 case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS: 354 ┆ /* 32 slots, no hotplug, only accepts pcie-root-port or 355 ┆ ┆* dmi-to-pci-bridge 356 ┆ ┆*/ 357 ┆ bus->flags = (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT | 358 ┆ ┆ ┆ ┆ ┆ VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE); 359 ┆ bus->minSlot = 0; 360 ┆ bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST; 361 ┆ break; The above is consistent with pcie-root itself, which also doesn't support plugging a pcie-to-pci-bridge directly into it but requires a pcie-root <- pcie-root-port <- pcie-to-pci-bridge topology.
Now, I don't quite recall *why* that is the case - perhaps Laine does? - but I'm sure we had very compelling reasons O:-)
Umm, because Marcel told us that was the proper behavior ? Right, Marcel?
But it works in qemu: # /usr/libexec/qemu-kvm -machine q35 -m 1024 -device pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x4 -device pcie-pci-bridge,id=pci.250,bus=pci.1,addr=0x0 -device pcie-pci-bridge,id=pci.251,bus=pci.1,addr=0x1 -spice port=5902,addr=0.0.0.0,disable-ticketing /var/lib/libvirt/images/q35.qcow2 It should be noted that QEMU in general allows users to make pretty unwise device placement choices without so much of a warning, so I would not take the above as proof the pcie-to-pci-bridge should be allowed to plug into pcie-expander-bus (or pcie-root) directly :)
Hi Laine, On 10/17/18 8:29 PM, Laine Stump wrote:
On 10/17/2018 08:56 AM, Andrea Bolognani wrote:
On Wed, 2018-10-17 at 10:50 +0800, Han Han wrote:
In libvirt, I found pcie-expander-bus controller doesn't support pcie-to-pci-bridge and pcie-switch-upstream-port. [...] # virsh -k0 -K0 define /tmp/c.xml Aside: the -k and -K virsh options are documented as
-k | --keepalive-interval=NUM keepalive interval in seconds, 0 for disable -K | --keepalive-count=NUM number of possible missed keepalive messages
So -k0 disables keepalive entirely, making -K0 unnecessary :)
error: Failed to define domain from /tmp/c.xml error: XML error: The device at PCI address 0000:01:00.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pcie-to-pci-bridge. [...] In function virDomainPCIAddressBusSetModel, I find pcie-expander-bus only supports pcie-root-port and dmi-to-pci-bridge 353 case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS: 354 ┆ /* 32 slots, no hotplug, only accepts pcie-root-port or 355 ┆ ┆* dmi-to-pci-bridge 356 ┆ ┆*/ 357 ┆ bus->flags = (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT | 358 ┆ ┆ ┆ ┆ ┆ VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE); 359 ┆ bus->minSlot = 0; 360 ┆ bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST; 361 ┆ break; The above is consistent with pcie-root itself, which also doesn't support plugging a pcie-to-pci-bridge directly into it but requires a pcie-root <- pcie-root-port <- pcie-to-pci-bridge topology.
Now, I don't quite recall *why* that is the case - perhaps Laine does? - but I'm sure we had very compelling reasons O:-) Umm, because Marcel told us that was the proper behavior ? Right, Marcel?
Yes it is. The pcie-pxb expander does not support "embedded" devices, any devices should be plugged into pcie ports. Even if QEMU cmd line allows embedded devices, it is not a supported configuration. Zooming it into the specific case of the pcie-pci bridge, this is a PCI Express device, and PCI Express devices may miss-behave if plugged as Integrated Devices. Thanks, Marcel
But it works in qemu: # /usr/libexec/qemu-kvm -machine q35 -m 1024 -device pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x4 -device pcie-pci-bridge,id=pci.250,bus=pci.1,addr=0x0 -device pcie-pci-bridge,id=pci.251,bus=pci.1,addr=0x1 -spice port=5902,addr=0.0.0.0,disable-ticketing /var/lib/libvirt/images/q35.qcow2 It should be noted that QEMU in general allows users to make pretty unwise device placement choices without so much of a warning, so I would not take the above as proof the pcie-to-pci-bridge should be allowed to plug into pcie-expander-bus (or pcie-root) directly :)
On Wed, Oct 17, 2018 at 8:56 PM Andrea Bolognani <abologna@redhat.com> wrote:
In libvirt, I found pcie-expander-bus controller doesn't support
On Wed, 2018-10-17 at 10:50 +0800, Han Han wrote: pcie-to-pci-bridge and pcie-switch-upstream-port. [...]
# virsh -k0 -K0 define /tmp/c.xml
Aside: the -k and -K virsh options are documented as
-k | --keepalive-interval=NUM keepalive interval in seconds, 0 for disable -K | --keepalive-count=NUM number of possible missed keepalive messages
So -k0 disables keepalive entirely, making -K0 unnecessary :)
Thanks for your advice.
error: Failed to define domain from /tmp/c.xml error: XML error: The device at PCI address 0000:01:00.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pcie-to-pci-bridge. [...] In function virDomainPCIAddressBusSetModel, I find pcie-expander-bus only supports pcie-root-port and dmi-to-pci-bridge 353 case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS: 354 ┆ /* 32 slots, no hotplug, only accepts pcie-root-port or 355 ┆ ┆* dmi-to-pci-bridge 356 ┆ ┆*/ 357 ┆ bus->flags = (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT | 358 ┆ ┆ ┆ ┆ ┆ VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE); 359 ┆ bus->minSlot = 0; 360 ┆ bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST; 361 ┆ break;
The above is consistent with pcie-root itself, which also doesn't support plugging a pcie-to-pci-bridge directly into it but requires a pcie-root <- pcie-root-port <- pcie-to-pci-bridge topology.
Now, I don't quite recall *why* that is the case - perhaps Laine does? - but I'm sure we had very compelling reasons O:-)
But it works in qemu: # /usr/libexec/qemu-kvm -machine q35 -m 1024 -device pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x4 -device pcie-pci-bridge,id=pci.250,bus=pci.1,addr=0x0 -device pcie-pci-bridge,id=pci.251,bus=pci.1,addr=0x1 -spice port=5902,addr=0.0.0.0,disable-ticketing /var/lib/libvirt/images/q35.qcow2
It should be noted that QEMU in general allows users to make pretty unwise device placement choices without so much of a warning, so I would not take the above as proof the pcie-to-pci-bridge should be allowed to plug into pcie-expander-bus (or pcie-root) directly :)
If pcie<-pcie-expander-bus<-(pcie-to-pci-bridge | pcie-switch-upstream-port) is an unwise device placement, we'd better to add the reason to code comments or commit msg or documents.
-- Andrea Bolognani / Red Hat / Virtualization
-- Best regards, ----------------------------------- Han Han Quality Engineer Redhat. Email: hhan@redhat.com Phone: +861065339333
participants (4)
-
Andrea Bolognani -
Han Han -
Laine Stump -
Marcel Apfelbaum