Re: [libvirt] How should libvirt apps enable virtio-pci for aarch64?
On 12/07/2015 03:19 AM, Pavel Fedin wrote:
Hello!
> - The PCIe controller XML is:
> <controller type='pci' index='0'
model='pcie-root'/>
> <controller type='pci' index='1'
model='dmi-to-pci-bridge'/>
> <controller type='pci' index='2'
model='pci-bridge'/>
> I have no idea if that's always going to be the expected XML, maybe it's not
> wise to hardcode that in apps.
Since we are discussing this, i have a question.
Why do we construct exactly this thing? What is "dmi-to-pci-bridge" and why do
we need it? I guess this is something PC-specific,
may be this layout has been copied from some real PC model, but i don't see any
practical sense in it.
This is likely just a side effect of the libvirt code requesting PCIe for
aarch64, but the original PCIe support was added for the x86 q35 layout.
Also, there are two problems with "pci-bridge":
1. Live migration of this thing in qemu is broken. After migration the bridge screws up,
lspci says "invalid header", and i don't
know whether it actually works because i never attach anything behind it, because of (2).
I didn't even know aarch64 migration was working...
2. After pcie-root we have PCI-X, which supports MSI-X. And after
pci-bridge we seem to have a plain PCI, which supports only plain
MSI. The problem here is that virtio seems to work only with MSI-X in any advanced mode
(multiqueue, vhost, etc). If i place it
behind the bridge (and default libvirt's logic is to place the device there), MSI-X
will not work. The same applies to passthrough
VFIO devices. This is especially painful because on real-life ARM64 platforms builtin
hardware seems to mandate MSI-X. For example
on ThunderX NIC driver simply does not support anything except MSI-X.
Maybe this is just a factor of libvirt specifying the wrong bits on the
aarch64 command line. Do you have a working qemu commandline outside of libvirt?
> * Next idea: Users specify something like like <address
type='pci'/> and
> libvirt fills in the address for us.
I like this one, and IMHO this would be nice to have regardless of the default. Manual
assignment of PCI layout is a tedious
process, which is not always necessary. I think it is quite logical to allow the user
just to say: "I want this device on the PCI
bus", and do the rest for him.
Agreed, I'll look into it in addition to the user PCI controller bits.
Also, this would allow to have a simple "Address type"
switch in e.g. virt-manager, so that i would be able to easily choose a way
which the device uses to connect to the system.
Currently I don't see a compelling reason to add this in the UI really. Sure
it's probably useful for virt dev testing to switching between mmio and pci
but long term everyone is going to use pci so I don't think end users are
going to be tweaking this. I think extending virt-xml to handle address bits
should cover it.
Thanks,
Cole