Hello! > - The PCIe controller XML is: > <controller type='pci' index='0' model='pcie-root'/> > <controller type='pci' index='1' model='dmi-to-pci-bridge'/> > <controller type='pci' index='2' model='pci-bridge'/> > I have no idea if that's always going to be the expected XML, maybe it's not > wise to hardcode that in apps. Since we are discussing this, i have a question. Why do we construct exactly this thing? What is "dmi-to-pci-bridge" and why do we need it? I guess this is something PC-specific, may be this layout has been copied from some real PC model, but i don't see any practical sense in it.

Also, there are two problems with "pci-bridge": 1. Live migration of this thing in qemu is broken. After migration the bridge screws up, lspci says "invalid header", and i don't know whether it actually works because i never attach anything behind it, because of (2).

2. After pcie-root we have PCI-X, which supports MSI-X. And after pci-bridge we seem to have a plain PCI, which supports only plain MSI. The problem here is that virtio seems to work only with MSI-X in any advanced mode (multiqueue, vhost, etc). If i place it behind the bridge (and default libvirt's logic is to place the device there), MSI-X will not work. The same applies to passthrough VFIO devices. This is especially painful because on real-life ARM64 platforms builtin hardware seems to mandate MSI-X. For example on ThunderX NIC driver simply does not support anything except MSI-X.

> * Next idea: Users specify something like like <address type='pci'/> and > libvirt fills in the address for us. I like this one, and IMHO this would be nice to have regardless of the default. Manual assignment of PCI layout is a tedious process, which is not always necessary. I think it is quite logical to allow the user just to say: "I want this device on the PCI bus", and do the rest for him.

Also, this would allow to have a simple "Address type" switch in e.g. virt-manager, so that i would be able to easily choose a way which the device uses to connect to the system.

For AARCH64, though... well, if you want to know why it's added for that machinetype, I guess you'd need to talk to the person who turned on addPCIeRoot for AARCH64 :-).

I actually wondered about that recently when I was tinkering with auto-adding USB2 controllers when machinetype is Q35 (i.e. "why are we adding an Intel/x86-specific controller to AARCH64 machines?")

> I didn't even know aarch64 migration was working...

Is this only a problem on aarch64, or is there a migration problem with pci-bridge on x86 as well? (It's possible there is but it hasn't been noticed, because pci-bridge likely isn't used much outside of q35 machines, and q35 was prohibited from migrating until qemu 2.4 due to the embedded sata driver which didn't support migration.)

BTW, does the aarch64 virt machinetype support any controller aside from the embedded pcie-root?

Right now we will auto-add only a pci-bridge if no available slot is found for a pci device, but we will (should anyway) auto-assign a slot on an *existing* PCIe controller if the device has PCIE as the preferred slot type.

> I don't remember the exact outcome. So, i decided to use addPCIeRoot instead, and everything just worked. Except the extra stuff. The origin of the problem was in my overloading "addPCIeRoot" to indicate that the other two controllers should also be added. You only made the mistake of thinking that the name of the variable was actually an accurate/complete description of what it did :-)

But apparently qemu is accepting "-device i82801b11-bridge" and -device pci-bridge, since that's what you get from libvirt. If these devices aren't supported for aarch64, they should be disabled in qemu (and not listed in the devices when libvirt asks).

Hi all,

Here are some possible solutions: * Drop the current behavior of adding a PCIe controller unconditionally, and instead require apps to specify it in the XML. Then, if libvirt sees a PCIe controller in the XML, default the virtio address type to pci. Apps will know if the OS they are installing supports virtio-pci (eventually via libosinfo), so this is the way we can implicitly ask libvirt 'allocate us pci addresses'

Upsides: - Solves both the stated problems. - Simplest addition for applications IMO Downsides: - Requires a libvirt behavior change, no longer adding the PCIe controller by default. But in practice I don't think it will really affect anyone, since there isn't really any OS support for virtio-pci yet, and no apps support it either AFAIK. - The PCIe controller is not strictly about virtio-pci, it's for enabling plain emulated PCI devices as well. So there is a use case for using the PCIe controller for a graphics card even while your OS doesn't yet support virtio-pci. In the big picture though this is a small time window with current OS, and users can work around it by manually requesting <address type='virtio-mmio'/>, so medium/long term this isn't a big deal IMO - The PCIe controller XML is: <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'/> <controller type='pci' index='2' model='pci-bridge'/> I have no idea if that's always going to be the expected XML, maybe it's not wise to hardcode that in apps. Laine?

* Next idea: Users specify something like like <address type='pci'/> and libvirt fills in the address for us.

Hi all, I'm trying to figure out how apps should request virtio-pci for libvirt + qemu + arm/aarch64. Let me provide some background. qemu's arm/aarch64 original virtio support is via virtio-mmio, libvirt XML <address type='virtio-mmio'/>. Currently this is what libvirt sets as the address default for all arm/aarch64 virtio devices in the XML. Long term though all arm virt will likely be using virtio-pci: it's faster, enables hotplug, is more x86 like, etc. Support for virtio-pci is newer and not as widespread. qemu has had the necessary support since 2.4 at least, but the guest side isn't well distributed yet. For example, Fedora 23 and earlier don't work out of the box with virtio-pci. Internal RHELSA (RHEL Server for Aarch64) builds have it recently working AFAIK. Libvirt has some support for enabling virtio-pci with aarch64, commits added by Pavel Fedin in v1.2.19. (See e8d55172544c1fafe31a9e09346bdebca4f0d6f9). The patches add a PCIe controller automatically to the XML (and qemu commandline) if qemu-system-aarch64 supports it. However virtio-mmio is still used as the default virtio address, given the current lack of OS support. So we are at the point where libvirt apps want to enable this, but presently there isn't a good solution; the only option is to fully allocate <address type='pci' ...> for each virtio device in the XML. This is suboptimal for 2 reasons: #1) apps need to duplicate libvirt's non-trivial address type=pci allocation logic #2) apps have to add an <address> block for every virtio device, which is less friendly than the x86 case where this is rarely required. Any XML device snippets that work for x86 likely won't give the desired result for aarch64, since they will default to virtio-mmio. Think virsh attach-device/attach-disk commands

Here are some possible solutions: * Drop the current behavior of adding a PCIe controller unconditionally, and instead require apps to specify it in the XML. Then, if libvirt sees a PCIe controller in the XML, default the virtio address type to pci. Apps will know if the OS they are installing supports virtio-pci (eventually via libosinfo), so this is the way we can implicitly ask libvirt 'allocate us pci addresses'

Upsides: - Solves both the stated problems. - Simplest addition for applications IMO Downsides: - Requires a libvirt behavior change, no longer adding the PCIe controller by default. But in practice I don't think it will really affect anyone, since there isn't really any OS support for virtio-pci yet, and no apps support it either AFAIK. - The PCIe controller is not strictly about virtio-pci, it's for enabling plain emulated PCI devices as well. So there is a use case for using the PCIe controller for a graphics card even while your OS doesn't yet support virtio-pci. In the big picture though this is a small time window with current OS, and users can work around it by manually requesting <address type='virtio-mmio'/>, so medium/long term this isn't a big deal IMO - The PCIe controller XML is: <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'/> <controller type='pci' index='2' model='pci-bridge'/> I have no idea if that's always going to be the expected XML, maybe it's not wise to hardcode that in apps. Laine? * Next idea: Users specify something like like <address type='pci'/> and libvirt fills in the address for us. Upsides: - We can stick with the current PCIe controller default and avoid some of the problems mentioned above. - An auto address feature may be useful in other contexts as well. Downsides: - Seems potentially tricky to implement in libvirt code. There's many places that check type=pci and key off that, seems like it would be easy to miss updating a check and cause regressions. Maybe we could add a new type like auto-pci to make it explicit. There's probably some implementation trick to make this safe, but at first glance it looked a little dicey.

- Doesn't really solve problem #2 mentioned up above... maybe we could change the address allocation logic to default to virtio-pci if there's already a virtio-pci device in the XML. But it's more work. - More work for apps, but nothing horrible.

* Change the default address type from virtio-mmio to pci, if qemu supports it. I'm listing this for completeness. In the short term this doesn't make sense as there isn't any OS releases that will work with this default. However it might be worth considering for the future, maybe keying off a particular qemu version or machine type. I suspect 2 years from now no one is going to be using virtio-mmio so long term it's not an ideal default.

I think the first option is best (keying off the PCIe controller specified by the user), with a longer term plan to change the default from mmio to pci. But I'm not really sold on anything either way. So I'm interested if anyone else has ideas.

On 12/07/2015 07:27 AM, Daniel P. Berrange wrote: > On Sun, Dec 06, 2015 at 09:46:56PM -0500, Cole Robinson wrote: >> Hi all, >> >> I'm trying to figure out how apps should request virtio-pci for libvirt + qemu >> + arm/aarch64. Let me provide some background. >> >> qemu's arm/aarch64 original virtio support is via virtio-mmio, libvirt XML >> <address type='virtio-mmio'/>. Currently this is what libvirt sets as the >> address default for all arm/aarch64 virtio devices in the XML. Long term >> though all arm virt will likely be using virtio-pci: it's faster, enables >> hotplug, is more x86 like, etc. >> >> Support for virtio-pci is newer and not as widespread. qemu has had the >> necessary support since 2.4 at least, but the guest side isn't well >> distributed yet. For example, Fedora 23 and earlier don't work out of the box >> with virtio-pci. Internal RHELSA (RHEL Server for Aarch64) builds have it >> recently working AFAIK. >> >> Libvirt has some support for enabling virtio-pci with aarch64, commits added >> by Pavel Fedin in v1.2.19. (See e8d55172544c1fafe31a9e09346bdebca4f0d6f9). The >> patches add a PCIe controller automatically to the XML (and qemu commandline) >> if qemu-system-aarch64 supports it. However virtio-mmio is still used as the >> default virtio address, given the current lack of OS support. >> >> So we are at the point where libvirt apps want to enable this, but presently >> there isn't a good solution; the only option is to fully allocate <address >> type='pci' ...> for each virtio device in the XML. This is suboptimal for 2 >> reasons: >> >> #1) apps need to duplicate libvirt's non-trivial address type=pci allocation logic >> >> #2) apps have to add an <address> block for every virtio device, which is less >> friendly than the x86 case where this is rarely required. Any XML device >> snippets that work for x86 likely won't give the desired result for aarch64, >> since they will default to virtio-mmio. Think virsh attach-device/attach-disk >> commands > > Yeah this is very undesirable for a default out of the box config - we should > always strive to "do the best thing" when no address is given. > >> Here are some possible solutions: >> >> * Drop the current behavior of adding a PCIe controller unconditionally, and >> instead require apps to specify it in the XML. Then, if libvirt sees a PCIe >> controller in the XML, default the virtio address type to pci. Apps will know >> if the OS they are installing supports virtio-pci (eventually via libosinfo), >> so this is the way we can implicitly ask libvirt 'allocate us pci addresses' > > Yes, clearly we need to record in libosinfo whether an OS can do PCI vs > MMIO. > >> Upsides: >> - Solves both the stated problems. >> - Simplest addition for applications IMO >> >> Downsides: >> - Requires a libvirt behavior change, no longer adding the PCIe controller by >> default. But in practice I don't think it will really affect anyone, since >> there isn't really any OS support for virtio-pci yet, and no apps support it >> either AFAIK. >> - The PCIe controller is not strictly about virtio-pci, it's for enabling >> plain emulated PCI devices as well. So there is a use case for using the PCIe >> controller for a graphics card even while your OS doesn't yet support >> virtio-pci. In the big picture though this is a small time window with current >> OS, and users can work around it by manually requesting <address >> type='virtio-mmio'/>, so medium/long term this isn't a big deal IMO >> - The PCIe controller XML is: >> <controller type='pci' index='0' model='pcie-root'/> >> <controller type='pci' index='1' model='dmi-to-pci-bridge'/> >> <controller type='pci' index='2' model='pci-bridge'/> >> I have no idea if that's always going to be the expected XML, maybe it's not >> wise to hardcode that in apps. Laine? >> >> >> * Next idea: Users specify something like like <address type='pci'/> and >> libvirt fills in the address for us. >> >> Upsides: >> - We can stick with the current PCIe controller default and avoid some of the >> problems mentioned above. >> - An auto address feature may be useful in other contexts as well. >> >> Downsides: >> - Seems potentially tricky to implement in libvirt code. There's many places >> that check type=pci and key off that, seems like it would be easy to miss >> updating a check and cause regressions. Maybe we could add a new type like >> auto-pci to make it explicit. There's probably some implementation trick to >> make this safe, but at first glance it looked a little dicey. > > I'm not sure it is actually all that hairy - it might be as simple as > updating only qemuAssignDevicePCISlots so that instread of: > > if (def->controllers[i]->info.type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_NONE) > continue; > > It handles type=pci with no values set too > I think my fear was that there are other places in domain_conf that check for ADDRESS_TYPE_PCI before we even get to assigning PCI slots. But i'll poke at it.