Hi Cole && All, Nice meeting you in the libvirt mail-list. Greetings from Linaro !!! I am Kevin Zhao from Linaro and working for OpenStack on Aarch64. Nowadays I find that the default "virtio" bus is "virtio-mmio" , not "virtio-pci". Since virtio-mmio do not has the host plugged function , something is wrong with the function in OpenStack Nova. Then I search and find some mail information as belows by Cole :-) 1) https://www.redhat.com/archives/libvir-list/2015-December/msg00217.html 2) https://www.redhat.com/archives/libvir-list/2016-March/msg00167.html I see some efforts have been doing by you. Really thanks for your efforts for Aarch64. And I have some small questions. 1. Is Libvirt planning to replace the default virtio-mmio to virtio-pci for Aarch64?

2. If not , how can I change the xml file(generated by Virsh) for virtio-pci so that Qemu can recognize it ?

Hi Cole, Thanks for your kindly help. On 13 June 2016 at 06:21, Cole Robinson <crobinso(a)redhat.com <mailto:crobinso@redhat.com>> wrote: On 06/12/2016 10:29 AM, Kevin Zhao wrote: > Hi Cole && All, > Nice meeting you in the libvirt mail-list. Greetings from Linaro !!! > I am Kevin Zhao from Linaro and working for OpenStack on Aarch64. > Nowadays I find that the default "virtio" bus is "virtio-mmio" , not > "virtio-pci". Since virtio-mmio do not has the host plugged function , > something is wrong with the function in OpenStack Nova. > > Then I search and find some mail information as belows by Cole :-) > 1) https://www.redhat.com/archives/libvir-list/2015-December/msg00217.html > 2) https://www.redhat.com/archives/libvir-list/2016-March/msg00167.html > > I see some efforts have been doing by you. Really thanks for your > efforts for Aarch64. And I have some small questions. > 1. Is Libvirt planning to replace the default virtio-mmio to virtio-pci > for Aarch64? We have vague plans to maybe change the virtio-mmio default to virtio-pci at some future point, based on some new enough qemu machine type. But the main blocker is that most linux distributions currently don't work out of the box with virtio-pci, at least Fedora doesn't as far as I know. So changing the default now is pre-mature OK . Hope it will be true in the future. > 2. If not , how can I change the xml file(generated by Virsh) for > virtio-pci so that Qemu can recognize it ? > The latest libvirt-1.3.5 release accepts device XML containing <address type='pci'/> . The address block is an explicit request to libvirt to 'allocate a PCI address' rather than the default virtio-mmio. So for example if you want to specify a virtio nic, but have it use virtio-pci rather than virtio-mmio, you can do something like: <interface type='network'> <source network='default'/> <model type='virtio'/> <address type='pci'/> </interface> But there isn't currently any switch to say 'always give me virtio-pci instead of virtio-mmio' In my opinion, that is to say, when I want to attach a block device to a guest and hope it take effects soon,*the only way is using 'scsi' instead of 'virtio'* in Aarch64 just like what virt-manager does since scsi supports hot plugged. Am I right?

On 06/12/2016 10:29 AM, Kevin Zhao wrote: > Hi Cole && All, > Nice meeting you in the libvirt mail-list. Greetings from Linaro !!! > I am Kevin Zhao from Linaro and working for OpenStack on Aarch64. > Nowadays I find that the default "virtio" bus is "virtio-mmio" , not > "virtio-pci". Since virtio-mmio do not has the host plugged function , > something is wrong with the function in OpenStack Nova. > > Then I search and find some mail information as belows by Cole :-) > 1) https://www.redhat.com/archives/libvir-list/2015-December/ msg00217.html > 2) https://www.redhat.com/archives/libvir-list/2016-March/ msg00167.html > > I see some efforts have been doing by you. Really thanks for your > efforts for Aarch64. And I have some small questions. > 1. Is Libvirt planning to replace the default virtio-mmio to virtio-pci > for Aarch64? We have vague plans to maybe change the virtio-mmio default to virtio-pci at some future point, based on some new enough qemu machine type. But the main blocker is that most linux distributions currently don't work out of the box with virtio-pci, at least Fedora doesn't as far as I know. So changing the default now is pre-mature

2. If not , how can I change the xml file(generated by Virsh) for > virtio-pci so that Qemu can recognize it ? > The latest libvirt-1.3.5 release accepts device XML containing <address type='pci'/> . The address block is an explicit request to libvirt to 'allocate a PCI address' rather than the default virtio-mmio. So for example if you want to specify a virtio nic, but have it use virtio-pci rather than virtio-mmio, you can do something like: <interface type='network'> <source network='default'/> <model type='virtio'/> <address type='pci'/> </interface>

But there isn't currently any switch to say 'always give me virtio-pci instead of virtio-mmio' - Cole

> I have changed the libvirt to 1.3.5 now, also add the pci to net-device xml like:  >  <address type='pci'/>,then use the virsh to boot the VM,the total xml file is: >  https://paste.fedoraproject.org/409534/71434141/ >  > After booting, the eth0 device disappear(eth0 occur when the address is > virtio-mmio), > but I can find another net-device, also it can't work for dhcp:  > 2: enp2s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state > UP group default qlen 1000 >     link/ether 52:54:00:0d:25:26 brd ff:ff:ff:ff:ff:ff >     inet6 fe80::5054:ff:fe0d:2526/64 scope link  >        valid_lft forever preferred_lft forever > Running lspci: > 02:01.0 Ethernet controller: Red Hat, Inc Virtio network device >  > My question is:  > 1. I use Debian 8 AArch64 as the Guest OS, do you think the virtio-pci for > net-device works is OK ?   My guess is that the support isn't complete. Last I checked Fedora doesn't even work with aarch64 virtio-pci + uefi, because it requires some kernel changes that haven't been fully upstreamed yet. But that was several months ago... There may be a way to work around it nowadays but I don't personally know. You may want to test with either RHELSA if you have a copy, or linaro images.   > 2. If I change the disk address-type to pci(Libvirt pass the virtio pci > parameters to Qemu for disk device), but I can't boot  > the VM. Does Qemu not support virtio pci for  disk device in AArch64?   That should work fine in my testing with RHELSA, so I don't think it's a libvirt or qemu limitation. Probably the guest OS + UEFI.

On Wed, 2016-08-17 at 09:18 -0400, Cole Robinson wrote: > > I have changed the libvirt to 1.3.5 now, also add the pci to net-device xml like:  > >  <address type='pci'/>,then use the virsh to boot the VM,the total xml file is: > >  https://paste.fedoraproject.org/409534/71434141/ > >  > > After booting, the eth0 device disappear(eth0 occur when the address is > > virtio-mmio), > > but I can find another net-device, also it can't work for dhcp:  > > 2: enp2s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state > > UP group default qlen 1000 > >     link/ether 52:54:00:0d:25:26 brd ff:ff:ff:ff:ff:ff > >     inet6 fe80::5054:ff:fe0d:2526/64 scope link  > >        valid_lft forever preferred_lft forever > > Running lspci: > > 02:01.0 Ethernet controller: Red Hat, Inc Virtio network device > >  > > My question is:  > > 1. I use Debian 8 AArch64 as the Guest OS, do you think the virtio-pci for > > net-device works is OK ? >  > My guess is that the support isn't complete. Last I checked Fedora doesn't > even work with aarch64 virtio-pci + uefi, because it requires some kernel > changes that haven't been fully upstreamed yet. But that was several months > ago... There may be a way to work around it nowadays but I don't personally > know. You may want to test with either RHELSA if you have a copy, or linaro > images. >  > > 2. If I change the disk address-type to pci(Libvirt pass the virtio pci > > parameters to Qemu for disk device), but I can't boot  > > the VM. Does Qemu not support virtio pci for  disk device in AArch64? >  > That should work fine in my testing with RHELSA, so I don't think it's a > libvirt or qemu limitation. Probably the guest OS + UEFI. Looks like Debian 8.5 simply doesn't include virtio-pci support:   $ lsinitramfs /boot/initrd.img | grep virtio   lib/modules/3.16.0-4-arm64/kernel/drivers/scsi/virtio_scsi.ko   lib/modules/3.16.0-4-arm64/kernel/drivers/block/virtio_blk.ko   lib/modules/3.16.0-4-arm64/kernel/drivers/virtio   lib/modules/3.16.0-4-arm64/kernel/drivers/virtio/virtio_ring.ko   lib/modules/3.16.0-4-arm64/kernel/drivers/virtio/virtio_mmio.ko   lib/modules/3.16.0-4-arm64/kernel/drivers/virtio/virtio.ko   lib/modules/3.16.0-4-arm64/kernel/drivers/net/virtio_net.ko   $ modprobe virtio-pci   modprobe: FATAL: Module virtio-pci not found. I think kernel 3.16.0 is simply too old.

On Fri, 2016-08-19 at 10:27 +0200, Guido Günther wrote: > > I think kernel 3.16.0 is simply too old. >  > According to the changelog it has been enabled in 3.14.7-1 on all architectures: >  >     http://metadata.ftp-master.debian.org/changelogs//main/l/linux/stable_changelog Are you referring to the   * [armhf] Enable VIRTIO_BALLOON and VIRTIO_PCI (Closes: #750742) entry? That's armhf, not arm64. Or was there a different reference that I missed?

Anyway, in my 8.5 arm64 guest, virtio-pci doesn't seem to be there at all:   $ grep VIRTIO /boot/config-3.16.0-4-arm64   CONFIG_NET_9P_VIRTIO=m   CONFIG_VIRTIO_BLK=m   CONFIG_SCSI_VIRTIO=m   CONFIG_VIRTIO_NET=m   CONFIG_VIRTIO_CONSOLE=m   CONFIG_HW_RANDOM_VIRTIO=m   CONFIG_VIRTIO=m   CONFIG_VIRTIO_BALLOON=m   CONFIG_VIRTIO_MMIO=m   # CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES is not set

On 08/17/2016 07:49 AM, Kevin Zhao wrote: > Hi Cole, > Long time no see~ > Thanks for giving help me about AArch64 of virtio-pci. You have rich > experience about this, > I am green hand and seeking for your help again :-) > Hello again, happy to help :) > On 13 June 2016 at 06:21, Cole Robinson <crobinso(a)redhat.com > <mailto:crobinso@redhat.com>> wrote: > > On 06/12/2016 10:29 AM, Kevin Zhao wrote: > > Hi Cole && All, > > Nice meeting you in the libvirt mail-list. Greetings from Linaro !!! > > I am Kevin Zhao from Linaro and working for OpenStack on Aarch64. > > Nowadays I find that the default "virtio" bus is "virtio-mmio" , not > > "virtio-pci". Since virtio-mmio do not has the host plugged function , > > something is wrong with the function in OpenStack Nova. > > > > Then I search and find some mail information as belows by Cole :-) > > 1) https://www.redhat.com/archives/libvir-list/2015- December/msg00217.html > <https://www.redhat.com/archives/libvir-list/2015- December/msg00217.html> > > 2) https://www.redhat.com/archives/libvir-list/2016- March/msg00167.html > <https://www.redhat.com/archives/libvir-list/2016- March/msg00167.html> > > > > I see some efforts have been doing by you. Really thanks for your > > efforts for Aarch64. And I have some small questions. > > 1. Is Libvirt planning to replace the default virtio-mmio to virtio-pci > > for Aarch64? > > We have vague plans to maybe change the virtio-mmio default to virtio-pci at > some future point, based on some new enough qemu machine type. But the main > blocker is that most linux distributions currently don't work out of the box > with virtio-pci, at least Fedora doesn't as far as I know. So changing the > default now is pre-mature > > > 2. If not , how can I change the xml file(generated by Virsh) for > > virtio-pci so that Qemu can recognize it ? > > > > The latest libvirt-1.3.5 release accepts device XML containing <address > type='pci'/> . The address block is an explicit request to libvirt to > 'allocate a PCI address' rather than the default virtio-mmio. So for example > if you want to specify a virtio nic, but have it use virtio-pci rather than > virtio-mmio, you can do something like: > > <interface type='network'> > <source network='default'/> > <model type='virtio'/> > <address type='pci'/> > </interface> > > > I have changed the libvirt to 1.3.5 now, also add the pci to net-device xml like: > <address type='pci'/>,then use the virsh to boot the VM,the total xml file is: > https://paste.fedoraproject.org/409534/71434141/ > > After booting, the eth0 device disappear(eth0 occur when the address is > virtio-mmio), > but I can find another net-device, also it can't work for dhcp: > 2: enp2s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state > UP group default qlen 1000 > link/ether 52:54:00:0d:25:26 brd ff:ff:ff:ff:ff:ff > inet6 fe80::5054:ff:fe0d:2526/64 scope link > valid_lft forever preferred_lft forever > Running lspci: > 02:01.0 Ethernet controller: Red Hat, Inc Virtio network device > > My question is: > 1. I use Debian 8 AArch64 as the Guest OS, do you think the virtio-pci for > net-device works is OK ? > My guess is that the support isn't complete. Last I checked Fedora doesn't even work with aarch64 virtio-pci + uefi, because it requires some kernel changes that haven't been fully upstreamed yet. But that was several months ago... There may be a way to work around it nowadays but I don't personally know. You may want to test with either RHELSA if you have a copy, or linaro images.

> 2. If I change the disk address-type to pci(Libvirt pass the virtio pci > parameters to Qemu for disk device), but I can't boot > the VM. Does Qemu not support virtio pci for disk device in AArch64? > That should work fine in my testing with RHELSA, so I don't think it's a libvirt or qemu limitation. Probably the guest OS + UEFI. - Cole > Big Big Thanks~ > > > But there isn't currently any switch to say 'always give me virtio-pci instead > of virtio-mmio' > > - Cole > >