Re: [libvirt] QEMU/KVM support for non zero PCI domain

On Apr 17, 2013, at 12:29 PM, Laine Stump <laine(a)laine.org&gt; wrote: > On 04/16/2013 11:23 AM, Charles Peri wrote: > >> Hi, >> I am trying to run Virt-Manager, using KVM on a SGIUV100 system with RHEL 6.1. I am trying to enable PCI pass-thorugh for the Mellanox, but the PCI domain for the device is 0001. >>> 4) Does libvirt actually get to the point of running qemu or kvm command? >>> 5) If the answer to (4) is "yes", can you send the qemu command that is >>> generated (it will be at the end of /var/log/libvirt/qemu/${guestname}.log) >> >> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name UV2-VM1-test1 -S -M rhel6.4.0 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -uuid e2428c26-f06a-b03c-c8d5-c528f2e85596 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/UV2-VM1-test1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/UV2-VM1-test1.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:5f:5f,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 > Okay. The important part here is this: > > -pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6 > > > This shows that libvirt is requesting to assign the device in domain 0, > bus 2, slot 0, function 4, > >>> 0001:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) >>> 0001:02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) >>> >>> When I try to add the device, I get an error stating that "Can't assign device inside non- zero PCI segment, as this KVM module doesn't support > I don't see this error in the above log, nor how it could have been > generated. There is nothing in the pci-assign device about a non-zero > domain. So I'm not sure how you arrived at that message. > > Laine, I was able to earlier install a later version of the libvirt on another system with RHEL 6.4, in which I got the error that "Can't assign device inside non-zero PCI segment, as this KVM module doesn't support" - I was not able to get back to that system, so could not get you the extant same configuration. But if I am able to reproduce it, I will send the config and message again. > At any rate, I've done some investigation and learned the following: > > 1) The qemu-0.12.x that is in RHEL6 only supports assigning devices in > domain 0. > > 2) "At some point" since then, the "host" option has been expanded to > allow including a domain id, e.g. "-device > pci-assign,host=0001:02:00.0,...", but that support was not backported > to RHEL. > > 3) Even if you were to have a version of qemu that supported assigning a > device in a non-zero domain, libvirt ignores the domain in the device > address, and only tells qemu about bus:slot.function. > > > So, libvirt should add support for assigning pci devices in non-0 > domains (and it's reasonably simple to do so), but it will only work on > newer versions of qemu than what you have, so you would have to build > your own (unsupported) upstream versions of both qemu and libvirt. > > Do we have the support built into version 1.4.0 of qemu?

Also, do I have to make the changes to the libvirt source code or is the support already available in some version, which I could use? Let me know, which versions or which source code will be a get place to start, if I want to add the support myself.