Hi Cole,
    Greate ! Thanks for your help.
    

On 17 August 2016 at 21:18, Cole Robinson <crobinso@redhat.com> wrote:
On 08/17/2016 07:49 AM, Kevin Zhao wrote:
> Hi Cole,
>      Long time no see~
>      Thanks for giving help me about AArch64 of virtio-pci. You have rich
> experience about this,
> I am green hand and seeking for your help again :-)
>

Hello again, happy to help :)

> On 13 June 2016 at 06:21, Cole Robinson <crobinso@redhat.com
> <mailto:crobinso@redhat.com>> wrote:
>
>     On 06/12/2016 10:29 AM, Kevin Zhao wrote:
>     > Hi Cole && All,
>     >      Nice meeting you in the libvirt mail-list. Greetings from Linaro !!!
>     >      I am Kevin Zhao from Linaro and working for OpenStack on Aarch64.
>     > Nowadays I find that the default "virtio" bus is "virtio-mmio" , not
>     > "virtio-pci". Since virtio-mmio do not has the host plugged function ,
>     > something is wrong with the function in OpenStack Nova.
>     >
>     >       Then I search and find some mail information as belows by Cole :-)
>     >       1) https://www.redhat.com/archives/libvir-list/2015-December/msg00217.html
>     <https://www.redhat.com/archives/libvir-list/2015-December/msg00217.html>
>     >       2) https://www.redhat.com/archives/libvir-list/2016-March/msg00167.html
>     <https://www.redhat.com/archives/libvir-list/2016-March/msg00167.html>
>     >
>     >        I see some efforts have been doing by you. Really thanks for your
>     > efforts for Aarch64. And I have some small questions.
>     >       1. Is Libvirt planning to replace the default virtio-mmio to virtio-pci
>     > for Aarch64?
>
>     We have vague plans to maybe change the virtio-mmio default to virtio-pci at
>     some future point, based on some new enough qemu machine type. But the main
>     blocker is that most linux distributions currently don't work out of the box
>     with virtio-pci, at least Fedora doesn't as far as I know. So changing the
>     default now is pre-mature
>
>     >       2. If not , how can I change the xml file(generated by Virsh) for
>     > virtio-pci  so that Qemu can recognize it ?
>     >
>
>     The latest libvirt-1.3.5 release accepts device XML containing <address
>     type='pci'/> . The address block is an explicit request to libvirt to
>     'allocate a PCI address' rather than the default virtio-mmio. So for example
>     if you want to specify a virtio nic, but have it use virtio-pci rather than
>     virtio-mmio, you can do something like:
>
>         <interface type='network'>
>           <source network='default'/>
>           <model type='virtio'/>
>           <address type='pci'/>
>         </interface>
>
>
> I have changed the libvirt to 1.3.5 now, also add the pci to net-device xml like:
>  <address type='pci'/>,then use the virsh to boot the VM,the total xml file is:
https://paste.fedoraproject.org/409534/71434141/
>
> After booting, the eth0 device disappear(eth0 occur when the address is
> virtio-mmio),
> but I can find another net-device, also it can't work for dhcp:
> 2: enp2s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
> UP group default qlen 1000
>     link/ether 52:54:00:0d:25:26 brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::5054:ff:fe0d:2526/64 scope link
>        valid_lft forever preferred_lft forever
> Running lspci:
> 02:01.0 Ethernet controller: Red Hat, Inc Virtio network device
>
> My question is:
> 1. I use Debian 8 AArch64 as the Guest OS, do you think the virtio-pci for
> net-device works is OK ?
>

My guess is that the support isn't complete. Last I checked Fedora doesn't
even work with aarch64 virtio-pci + uefi, because it requires some kernel
changes that haven't been fully upstreamed yet. But that was several months
ago... There may be a way to work around it nowadays but I don't personally
know. You may want to test with either RHELSA if you have a copy, or linaro
images.
You are right !  Today I happen to receive a RHEL7.3 for AArch64. I have been 
using this as Guest OS. It has virtio-pci driver. 
In Libvirt 1.3.5, I use the xml : https://paste.fedoraproject.org/410158/21672147/
Virtio-pci disk works fine(RHEL73+UEFI). But using virsh attach-device to attach
 a device:
<disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/rhel2.qcow2"/>
      <target dev="vdb" bus="virtio"/>
   <address type='pci'/>
</disk>
After adding, I can't find vdb in the Guest OS. I need to rescan the pci by: 
echo 1 > /sys/bus/pci/rescan
Then vdb appear. Do you have the same problem?

Another question, when changing to Libvirt 2.1.0, I add the : <address type='pci'/>
to a disk device or net device, it can't work, it seems that user need to assign
the PCI manually ?  It means that I should assign the slots and bus all by myself?

Thanks~

> 2. If I change the disk address-type to pci(Libvirt pass the virtio pci
> parameters to Qemu for disk device), but I can't boot
> the VM. Does Qemu not support virtio pci for  disk device in AArch64?
>

That should work fine in my testing with RHELSA, so I don't think it's a
libvirt or qemu limitation. Probably the guest OS + UEFI.

- Cole

> Big Big Thanks~
>
>
>     But there isn't currently any switch to say 'always give me virtio-pci instead
>     of virtio-mmio'
>
>     - Cole
>
>