On Thu, Feb 16, 2017 at 04:58:52PM +0100, Andrea Bolognani wrote:

On Wed, 2017-02-15 at 09:24 +0000, Daniel P. Berrange wrote: [...]

...

$ virsh start f22-arm32 error: Failed to start domain f22-arm32 error: internal error: qemu unexpectedly closed the monitor: 2017-02-15T09:24:03.967648Z qemu-system-arm: -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: MSI is not supported by interrupt controller 2017-02-15T09:24:03.968154Z qemu-system-arm: -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: Device initialization failed   I've not bisected it other than to find it works in 2.5.0 and is broken in 3.0.0

I'd like you to share a few additional details:

  * What does the guest XML look like after libvirt has     had a change to augment it?

<domain type='qemu'> <name>f22-arm32</name> <uuid>a6bc14fb-585b-40b0-a15b-5e19f26079ba</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='armv7l' machine='virt-2.7'>hvm</type> <boot dev='hd'/> </os> <features> <gic version='3'/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-arm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/berrange/VirtualMachines/demo.qcow2'/> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='ioh3420'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='ioh3420'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='ioh3420'/> <target chassis='3' port='0xa'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <interface type='user'> <mac address='52:54:00:87:d4:c0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> </devices> </domain>

  * What QEMU command line does libvirt generate based     on said guest XML?

LC_ALL=C PATH=/home/berrange/.local/bin:/home/berrange/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/berrange/.local/bin:/home/berrange/bin HOME=/home/berrange USER=berrange LOGNAME=berrange QEMU_AUDIO_DRV=none /usr/bin/qemu-system-arm -name guest=f22-arm32,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/home/berrange/.config/libvirt/qemu/lib/domain-1-f22-arm32/master-key.aes -machine virt-2.7,accel=tcg,usb=off,dump-guest-core=off,gic-version=3 -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid a6bc14fb-585b-40b0-a15b-5e19f26079ba -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/home/berrange/.config/libvirt/qemu/lib/domain-1-f22-arm32/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 -device ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 -device ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x0 -drive file=/home/berrange/VirtualMachines/demo.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev user,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:87:d4:c0,bus=pci.1,addr=0x0 -serial pty -msg timestamp=on char device redirected to /dev/pts/20 (label serial0) 2017-02-16T16:04:43.592233Z qemu-system-arm: -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: MSI is not supported by interrupt controller 2017-02-16T16:04:43.592725Z qemu-system-arm: -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: Device initialization failed 2017-02-16 16:04:43.595+0000: shutting down, reason=failed

  * What QEMU version are you running?

2017-02-16 16:04:43.517+0000: starting up libvirt version: 3.0.0, package: 1.fc25 (Unknown, 2017-01-19-10:16:54, t460), qemu version: 2.7.1(qemu-2.7.1-2.fc25), hostname: t460

I'm asking because I've just spent some time trying to run ARM guests[1] on my laptop[2] and I can't reproduce the failure you're describing, so I need more information to try and narrow it down.

my favourite type of bug :-)

---

I would expect such an error to pop up simply by running

  $ qemu-system-arm -nodefaults -M virt \     -device ioh3420,port=0x8,...

Does that happen on your system?

That gets a segmentation fault !

I didn't realize the mach-virt machine type was not an aarch64-only thing. Indeed, it's available through qemu-system-arm too, and the hardware seems to be the same, eg. running

  $ echo -e 'info qtree\nquit\n' | \     qemu-system-$arch -M virt -monitor stdio

yields the same output.

The only difference AFAICT is that qemu-system-arm limits the selection of CPUs to 32-bit models only, eg. cortex-a15 is available on both but only qemu-system-aarch64 lets you use cortex-a57.

Yep, makes sense - most of the arm code is shared in QEMU

I know 32-bit UEFI is a thing, because it's used on a bunch of budget x86 tablet and causes grief and pain to anyone trying to run Linux on them. However, Fedora only ships 64-bit binaries (edk2-ovmf and edk2-aarch64 packages) so I can't really try whether an armv7l guest can boot using UEFI.

Does -M Virt require UEFI ? I thought those were orthoganol choices.

Speaking of booting the guest, how would that work with the guest XML you're feeding libvirt, exactly? Since you don't have any sort of firmware, the only way I can see it working is to to have <kernel>, <initrd> and <cmdline> elements inside <domain><os>, and libvirt can't possibly figure out their values for you...

That is correct - it can't boot, it needs kernel/initrd. I'd be happy if QEMU actually got far enough to tell me I forgot to give it a kernel or firmware :-)

---

Does it help at all to use

  <address type='virtio-mmio'/>

in order to force the the SCSI controller and network interface to use virtio-mmio instead of virtio-pci?

Yes, that launches QEMU which then ultimately exits with a message [quote] Trying to execute code outside RAM or ROM at 0x08000000 This usually means one of the following happened: (1) You told QEMU to execute a kernel for the wrong machine type, and it crashed on startup (eg trying to run a raspberry pi kernel on a versatilepb QEMU machine) (2) You didn't give QEMU a kernel or BIOS filename at all, and QEMU executed a ROM full of no-op instructions until it fell off the end (3) Your guest kernel has a bug and crashed by jumping off into nowhere This is almost always one of the first two, so check your command line and that you are using the right type of kernel for this machine. If you think option (3) is likely then you can try debugging your guest with the -d debug options; in particular -d guest_errors will cause the log to include a dump of the guest register state at this point. Execution cannot continue; stopping here. [/quote] Clearly item (2) is the cause here - no kernel or firmware Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|

On Fri, 2017-02-17 at 15:56 +0100, Andrea Bolognani wrote:

libvirt is picking GICv3 here because QEMU reports it as a viable emulated GIC; however, as I understand it the emulated GICv3 doesn't have MSI support, and without that PCIe can't work. If you manually switch to GICv2 you should be able to run the guest succesfully.   We should find a way to detect whether the interrupt controller will support PCIe, and fall back to using virtio-mmio when it doesn't. Eric, any ideas about how we could achieve that?

Actually, we will probably want to do the opposite, eg. pick GICv2 over GICv3 if the latter doesn't allow us to use PCIe. --  Andrea Bolognani / Red Hat / Virtualization

On Sun, 2017-02-19 at 18:54 +0100, Auger Eric wrote:

...

...

We should find a way to detect whether the interrupt controller will support PCIe, and fall back to using virtio-mmio when it doesn't. Eric, any ideas about how we could achieve that?   Actually, we will probably want to do the opposite, eg. pick GICv2 over GICv3 if the latter doesn't allow us to use PCIe.   Yes I think the easiest solution is to select the GICv2 + v2m combo to

Not sure how the "combo" part works... I noticed that, if I start a guest with   -machine virt,gic-version=3 I only get one GIC-related device:   dev: arm-gicv3, id ""     gpio-out "sysbus-irq" 4     gpio-in "" 288     num-cpu = 1 (0x1)     num-irq = 288 (0x120)     revision = 3 (0x3)     has-security-extensions = false     mmio 0000000008000000/0000000000010000     mmio 00000000080a0000/0000000000020000 However, if I use gic-version=2, I get:   dev: arm-gicv2m, id ""     gpio-out "sysbus-irq" 64     base-spi = 48 (0x30)     num-spi = 64 (0x40)     mmio 0000000008020000/0000000000001000   dev: arm_gic, id ""     gpio-out "sysbus-irq" 4     gpio-in "" 288     num-cpu = 1 (0x1)     num-irq = 288 (0x120)     revision = 2 (0x2)     has-security-extensions = false     mmio 0000000008000000/0000000000001000     mmio 0000000008010000/0000000000002000     mmio ffffffffffffffff/0000000000000100 Is that what you're referring to? Do we need to care about the fact that it's two devices rather than one?

get the MSI support. This depends on whether the limitations linked to GICv2 usage are acceptable in your case (max 8 cpus for instance).

I think the advantages of using virtio-pci instead of virtio-mmio offset the limitations of GICv2, at least in the default case. Aside from not being able to have more than 8 vCPUs, what other limitations are we looking at? Anyway, my question was really: how can we tell whether a certain gic-version will support PCIe? Is there a way to ask QEMU for the information, or would we have to hardcode this knowledge in libvirt? The latter sounds like it would backfire quite easily, as I assume the emulated GICv3 not having PCIe support is only a matter of the relevant code not having been implemented *yet*, not something that's always going to be true. --  Andrea Bolognani / Red Hat / Virtualization