Re: [libvirt] ARMv7 guest PCI support broken in 3.0.0 onwards

On Wed, 2017-02-15 at 09:24 +0000, Daniel P. Berrange wrote: [...] ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: MSI is not supported by interrupt controller I'd like you to share a few additional details:   * What does the guest XML look like after libvirt has     had a change to augment it?   * What QEMU command line does libvirt generate based     on said guest XML?   * What QEMU version are you running? I'm asking because I've just spent some time trying to run ARM guests[1] on my laptop[2] and I can't reproduce the failure you're describing, so I need more information to try and narrow it down. After the skip, a bunch of unstructured rambling about stuff I've discovered in the process, interleaved with more questions about your issue. Have fun! :) --- I would expect such an error to pop up simply by running   $ qemu-system-arm -nodefaults -M virt \     -device ioh3420,port=0x8,... Does that happen on your system? --- I didn't realize the mach-virt machine type was not an aarch64-only thing. Indeed, it's available through qemu-system-arm too, and the hardware seems to be the same, eg. running   $ echo -e 'info qtree\nquit\n' | \     qemu-system-$arch -M virt -monitor stdio yields the same output. The only difference AFAICT is that qemu-system-arm limits the selection of CPUs to 32-bit models only, eg. cortex-a15 is available on both but only qemu-system-aarch64 lets you use cortex-a57. --- I know 32-bit UEFI is a thing, because it's used on a bunch of budget x86 tablet and causes grief and pain to anyone trying to run Linux on them. However, Fedora only ships 64-bit binaries (edk2-ovmf and edk2-aarch64 packages) so I can't really try whether an armv7l guest can boot using UEFI. --- Speaking of booting the guest, how would that work with the guest XML you're feeding libvirt, exactly? Since you don't have any sort of firmware, the only way I can see it working is to to have <kernel>, <initrd> and <cmdline> elements inside <domain><os>, and libvirt can't possibly figure out their values for you... --- Does it help at all to use   <address type='virtio-mmio'/> in order to force the the SCSI controller and network interface to use virtio-mmio instead of virtio-pci? --- As a reference, here's the XML for my Fedora 25/armv7l guest: <domain type='qemu'>   <name>f25-armv7l</name>   <uuid>029956e6-5e98-4ca6-8b2b-72322f73f3ea</uuid>   <memory unit='KiB'>2097152</memory>   <currentMemory unit='KiB'>2097152</currentMemory>   <vcpu placement='static'>1</vcpu>   <os>     <type arch='armv7l' machine='virt-2.9'>hvm</type>     <kernel>/home/abologna/.local/share/libvirt/images/f25-armv7l.vmlinuz</kernel>     <initrd>/home/abologna/.local/share/libvirt/images/f25-armv7l.initramfs</initrd>     <cmdline>console=ttyAMA0 rw root=LABEL=_/ rootwait</cmdline>     <boot dev='hd'/>   </os>   <features>     <gic version='2'/>   </features>   <cpu mode='custom' match='exact'>     <model fallback='allow'>cortex-a15</model>   </cpu>   <clock offset='utc'/>   <on_poweroff>destroy</on_poweroff>   <on_reboot>restart</on_reboot>   <on_crash>restart</on_crash>   <devices>     <emulator>/usr/local/bin/qemu-system-arm</emulator>     <disk type='file' device='disk'>       <driver name='qemu' type='qcow2'/>       <source file='/home/abologna/.local/share/libvirt/images/f25-armv7l.qcow2'/>       <target dev='sda' bus='scsi'/>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>     </disk>     <controller type='scsi' index='0' model='virtio-scsi'>       <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>     </controller>     <controller type='pci' index='0' model='pcie-root'/>     <controller type='pci' index='1' model='pcie-root-port'>       <model name='ioh3420'/>       <target chassis='1' port='0x8'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>     </controller>     <controller type='pci' index='2' model='pcie-root-port'>       <model name='ioh3420'/>       <target chassis='2' port='0x9'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>     </controller>     <controller type='pci' index='3' model='pcie-root-port'>       <model name='ioh3420'/>       <target chassis='3' port='0xa'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>     </controller>     <interface type='user'>       <mac address='52:54:00:9a:47:50'/>       <model type='virtio'/>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>     </interface>     <serial type='pty'>       <target port='0'/>     </serial>     <console type='pty'>       <target type='serial' port='0'/>     </console>   </devices> </domain> --- And here's that for my Fedora 25/aarch64 guest: <domain type='qemu'>   <name>f25-aarch64</name>   <uuid>e3e954ba-5280-4b6b-9bfc-7f944f2c67b2</uuid>   <memory unit='KiB'>2097152</memory>   <currentMemory unit='KiB'>2097152</currentMemory>   <vcpu placement='static'>1</vcpu>   <os>     <type arch='aarch64' machine='virt-2.9'>hvm</type>     <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader>     <nvram>/home/abologna/.config/libvirt/qemu/nvram/f25-aarch64_VARS.fd</nvram>     <boot dev='hd'/>   </os>   <features>     <gic version='2'/>   </features>   <cpu mode='custom' match='exact'>     <model fallback='allow'>cortex-a57</model>   </cpu>   <clock offset='utc'/>   <on_poweroff>destroy</on_poweroff>   <on_reboot>restart</on_reboot>   <on_crash>restart</on_crash>   <devices>     <emulator>/usr/local/bin/qemu-system-aarch64</emulator>     <disk type='file' device='disk'>       <driver name='qemu' type='qcow2'/>       <source file='/home/abologna/.local/share/libvirt/images/f25-aarch64.qcow2'/>       <target dev='sda' bus='scsi'/>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>     </disk>     <controller type='scsi' index='0' model='virtio-scsi'>       <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>     </controller>     <controller type='pci' index='0' model='pcie-root'/>     <controller type='pci' index='1' model='pcie-root-port'>       <model name='ioh3420'/>       <target chassis='1' port='0x8'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>     </controller>     <controller type='pci' index='2' model='pcie-root-port'>       <model name='ioh3420'/>       <target chassis='2' port='0x9'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>     </controller>     <controller type='pci' index='3' model='pcie-root-port'>       <model name='ioh3420'/>       <target chassis='3' port='0xa'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>     </controller>     <interface type='user'>       <mac address='52:54:00:c3:33:4a'/>       <model type='virtio'/>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>     </interface>     <serial type='pty'>       <target port='0'/>     </serial>     <console type='pty'>       <target type='serial' port='0'/>     </console>   </devices> </domain> [1] Fedora 25/armv7l, plus Fedora 25/aarch64 for comparison [2] Fedora 24/x86_64 with libvirt 3.0.0; I've tried, in turn,     QEMU master, QEMU 2.6 (as shipped with Fedora) and     upstream QEMU 2.5 --  Andrea Bolognani / Red Hat / Virtualization