[libvirt] QEMU/KVM support for non zero PCI domain
Hi, I am trying to run Virt-Manager, using KVM on a SGIUV100 system with RHEL 6.1. I am trying to enable PCI pass-thorugh for the Mellanox, but the PCI domain for the device is 0001. 0001:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) 0001:02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) When I try to add the device, I get an error stating that "Can't assign device inside non- zero PCI segment, as this KVM module doesn't support it". Is there a version of KVM module that supports non zero PCI segment? Sorry, if this is not the right email list to ask this question, I checked on the KVM IRC, but haven't got any answer. Thanks, Charles
On 04/12/2013 05:00 PM, Charles Peri wrote:
Hi, I am trying to run Virt-Manager, using KVM on a SGIUV100 system with RHEL 6.1. I am trying to enable PCI pass-thorugh for the Mellanox, but the PCI domain for the device is 0001.
Interesting! I knew it existed, but I've never seen hardware that actually had multiple domains.
0001:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) 0001:02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
When I try to add the device, I get an error stating that "Can't assign device inside non- zero PCI segment, as this KVM module doesn't support it". Is there a version of KVM module that supports non zero PCI segment?
Huh. I would have figured it would be stopped by libvirt first, as libvirt's PCI address parsing currently rejects anything with domain or bus that isn't 0. That doesn't sound like a libvirt error message though, which implies that, at least for host-side PCI addresses, libvirt is doing the right thing, and the error is coming back from kvm.
Sorry, if this is not the right email list to ask this question, I checked on the KVM IRC, but haven't got any answer.
A few questions: 1) What is your Linux distro and version 2) what version of qemu/kvm do you have 3) what version of libvirt? 4) Does libvirt actually get to the point of running qemu or kvm command? 5) If the answer to (4) is "yes", can you send the qemu command that is generated (it will be at the end of /var/log/libvirt/qemu/${guestname}.log)
Hi Laine, Here are the information you have requested: 1) What is your Linux distro and version RHEL 6.1 - 2.6.32-131.0.15.el6.x86_64 2) what version of qemu/kvm do you have [root@uv2 qemu]# /usr/libexec/qemu-kvm -version QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard 3) what version of libvirt? [root@uv2 qemu]# libvirtd --version libvirtd (libvirt) 0.10.2 4) Does libvirt actually get to the point of running qemu or kvm command? 5) If the answer to (4) is "yes", can you send the qemu command that is generated (it will be at the end of /var/log/libvirt/qemu/${guestname}.log) LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name UV2-VM1-test1 -S -M rhel6.4.0 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -uuid e2428c26-f06a-b03c-c8d5-c528f2e85596 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/UV2-VM1-test1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/UV2-VM1-test1.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:5f:5f,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 char device redirected to /dev/pts/2 get_real_device: /sys/bus/pci/devices/0000:02:00.4/resource: No such file or directory qemu-kvm: -device pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6: pci-assign: Error: Couldn't get real device (hostdev0)! qemu-kvm: -device pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6: Device 'pci-assign' could not be initialized 2013-04-15 19:38:31.454+0000: shutting down ******************** As you can see, I cannot run qemu command directly from libvirt. But if invoke qemu-kvm from command line: [root@uv2 VM_test]# /usr/libexec/qemu-kvm -smp 2 -hda rhel_62.img -m 8192 -redir tcp:52110::22 -device pci-assign,host=02:00.0 -net nic -net user get_real_device: /sys/bus/pci/devices/0000:02:00.0/config: No such file or directory qemu-kvm: -device pci-assign,host=02:00.0: pci-assign: Error: Couldn't get real device ((null))! qemu-kvm: -device pci-assign,host=02:00.0: Device 'pci-assign' could not be initialize [root@uv2 VM_test]# /usr/libexec/qemu-kvm -smp 2 -hda rhel_62.img -m 8192 -redir tcp:52110::22 -device pci-assign,host=0001:02:00.0 -net nic -net user qemu-kvm: -device pci-assign,host=0001:02:00.0: Property 'pci-assign.host' doesn't take value '0001:02:00.0' [root@uv2 VM_test]# /usr/libexec/qemu-kvm --version QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard ********* Installing version 1.4.0 of qemu: [root@uv2 VM_test]# /usr/local/bin/qemu-system-x86_64 -smp 2 -hda rhel_62.img -m 8192 -redir tcp:52110::22 -device pci-assign,host=02:00.0 -net nic -net user qemu-system-x86_64: -device pci-assign,host=02:00.0: pci-assign: error: requires KVM support qemu-system-x86_64: -device pci-assign,host=02:00.0: Device 'kvm-pci-assign' could not be initialized [root@uv2 VM_test]# /usr/local/bin/qemu-system-x86_64 --version QEMU emulator version 1.4.0, Copyright (c) 2003-2008 Fabrice Bellard [root@uv2 VM_test]# /usr/local/bin/qemu-system-x86_64 -smp 2 -hda rhel_62.img -m 8192 -redir tcp:52110::22 -device pci-assign,host=0001:02:00.0 -net nic -net user qemu-system-x86_64: -device pci-assign,host=0001:02:00.0: pci-assign: error: requires KVM support qemu-system-x86_64: -device pci-assign,host=0001:02:00.0: Device 'kvm-pci-assign' could not be initialized Any idea, how I could get support for 0001 pci domain? Thanks, Charles ----- Original Message ----- From: "Laine Stump" <laine@laine.org> To: libvir-list@redhat.com Sent: Friday, April 12, 2013 8:48:24 PM Subject: Re: [libvirt] QEMU/KVM support for non zero PCI domain On 04/12/2013 05:00 PM, Charles Peri wrote:
Hi, I am trying to run Virt-Manager, using KVM on a SGIUV100 system with RHEL 6.1. I am trying to enable PCI pass-thorugh for the Mellanox, but the PCI domain for the device is 0001.
Interesting! I knew it existed, but I've never seen hardware that actually had multiple domains.
0001:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) 0001:02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
When I try to add the device, I get an error stating that "Can't assign device inside non- zero PCI segment, as this KVM module doesn't support it". Is there a version of KVM module that supports non zero PCI segment?
Huh. I would have figured it would be stopped by libvirt first, as libvirt's PCI address parsing currently rejects anything with domain or bus that isn't 0. That doesn't sound like a libvirt error message though, which implies that, at least for host-side PCI addresses, libvirt is doing the right thing, and the error is coming back from kvm.
Sorry, if this is not the right email list to ask this question, I checked on the KVM IRC, but haven't got any answer.
A few questions: 1) What is your Linux distro and version 2) what version of qemu/kvm do you have 3) what version of libvirt? 4) Does libvirt actually get to the point of running qemu or kvm command? 5) If the answer to (4) is "yes", can you send the qemu command that is generated (it will be at the end of /var/log/libvirt/qemu/${guestname}.log) -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 04/16/2013 11:23 AM, Charles Peri wrote:
Hi, I am trying to run Virt-Manager, using KVM on a SGIUV100 system with RHEL 6.1. I am trying to enable PCI pass-thorugh for the Mellanox, but the PCI domain for the device is 0001.
4) Does libvirt actually get to the point of running qemu or kvm command?
5) If the answer to (4) is "yes", can you send the qemu command that is generated (it will be at the end of /var/log/libvirt/qemu/${guestname}.log)
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name UV2-VM1-test1 -S -M rhel6.4.0 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -uuid e2428c26-f06a-b03c-c8d5-c528f2e85596 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/UV2-VM1-test1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/UV2-VM1-test1.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:5f:5f,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Okay. The important part here is this: -pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6 This shows that libvirt is requesting to assign the device in domain 0, bus 2, slot 0, function 4,
0001:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) 0001:02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
When I try to add the device, I get an error stating that "Can't assign device inside non- zero PCI segment, as this KVM module doesn't support
I don't see this error in the above log, nor how it could have been generated. There is nothing in the pci-assign device about a non-zero domain. So I'm not sure how you arrived at that message. At any rate, I've done some investigation and learned the following: 1) The qemu-0.12.x that is in RHEL6 only supports assigning devices in domain 0. 2) "At some point" since then, the "host" option has been expanded to allow including a domain id, e.g. "-device pci-assign,host=0001:02:00.0,...", but that support was not backported to RHEL. 3) Even if you were to have a version of qemu that supported assigning a device in a non-zero domain, libvirt ignores the domain in the device address, and only tells qemu about bus:slot.function. So, libvirt should add support for assigning pci devices in non-0 domains (and it's reasonably simple to do so), but it will only work on newer versions of qemu than what you have, so you would have to build your own (unsupported) upstream versions of both qemu and libvirt.
On Apr 17, 2013, at 12:29 PM, Laine Stump <laine@laine.org> wrote:
On 04/16/2013 11:23 AM, Charles Peri wrote:
Hi, I am trying to run Virt-Manager, using KVM on a SGIUV100 system with RHEL 6.1. I am trying to enable PCI pass-thorugh for the Mellanox, but the PCI domain for the device is 0001.
4) Does libvirt actually get to the point of running qemu or kvm command?
5) If the answer to (4) is "yes", can you send the qemu command that is generated (it will be at the end of /var/log/libvirt/qemu/${guestname}.log)
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name UV2-VM1-test1 -S -M rhel6.4.0 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -uuid e2428c26-f06a-b03c-c8d5-c528f2e85596 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/UV2-VM1-test1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/UV2-VM1-test1.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:5f:5f,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Okay. The important part here is this:
-pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6
This shows that libvirt is requesting to assign the device in domain 0, bus 2, slot 0, function 4,
0001:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) 0001:02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
When I try to add the device, I get an error stating that "Can't assign device inside non- zero PCI segment, as this KVM module doesn't support
I don't see this error in the above log, nor how it could have been generated. There is nothing in the pci-assign device about a non-zero domain. So I'm not sure how you arrived at that message.
Laine, I was able to earlier install a later version of the libvirt on another system with RHEL 6.4, in which I got the error that "Can't assign device inside non-zero PCI segment, as this KVM module doesn't support" - I was not able to get back to that system, so could not get you the extant same configuration. But if I am able to reproduce it, I will send the config and message again.
At any rate, I've done some investigation and learned the following:
1) The qemu-0.12.x that is in RHEL6 only supports assigning devices in domain 0.
2) "At some point" since then, the "host" option has been expanded to allow including a domain id, e.g. "-device pci-assign,host=0001:02:00.0,...", but that support was not backported to RHEL.
3) Even if you were to have a version of qemu that supported assigning a device in a non-zero domain, libvirt ignores the domain in the device address, and only tells qemu about bus:slot.function.
So, libvirt should add support for assigning pci devices in non-0 domains (and it's reasonably simple to do so), but it will only work on newer versions of qemu than what you have, so you would have to build your own (unsupported) upstream versions of both qemu and libvirt.
Do we have the support built into version 1.4.0 of qemu? Also, do I have to make the changes to the libvirt source code or is the support already available in some version, which I could use? Let me know, which versions or which source code will be a get place to start, if I want to add the support myself. Thanks, Charles
On 04/18/2013 03:13 PM, Charles Peri wrote:
On Apr 17, 2013, at 12:29 PM, Laine Stump <laine@laine.org> wrote:
On 04/16/2013 11:23 AM, Charles Peri wrote:
Hi, I am trying to run Virt-Manager, using KVM on a SGIUV100 system with RHEL 6.1. I am trying to enable PCI pass-thorugh for the Mellanox, but the PCI domain for the device is 0001.
4) Does libvirt actually get to the point of running qemu or kvm command? 5) If the answer to (4) is "yes", can you send the qemu command that is generated (it will be at the end of /var/log/libvirt/qemu/${guestname}.log)
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name UV2-VM1-test1 -S -M rhel6.4.0 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -uuid e2428c26-f06a-b03c-c8d5-c528f2e85596 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/UV2-VM1-test1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/UV2-VM1-test1.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:5f:5f,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 Okay. The important part here is this:
-pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6
This shows that libvirt is requesting to assign the device in domain 0, bus 2, slot 0, function 4,
0001:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) 0001:02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
When I try to add the device, I get an error stating that "Can't assign device inside non- zero PCI segment, as this KVM module doesn't support I don't see this error in the above log, nor how it could have been generated. There is nothing in the pci-assign device about a non-zero domain. So I'm not sure how you arrived at that message.
Laine, I was able to earlier install a later version of the libvirt on another system with RHEL 6.4, in which I got the error that "Can't assign device inside non-zero PCI segment, as this KVM module doesn't support" - I was not able to get back to that system, so could not get you the extant same configuration. But if I am able to reproduce it, I will send the config and message again.
At any rate, I've done some investigation and learned the following:
1) The qemu-0.12.x that is in RHEL6 only supports assigning devices in domain 0.
2) "At some point" since then, the "host" option has been expanded to allow including a domain id, e.g. "-device pci-assign,host=0001:02:00.0,...", but that support was not backported to RHEL.
3) Even if you were to have a version of qemu that supported assigning a device in a non-zero domain, libvirt ignores the domain in the device address, and only tells qemu about bus:slot.function.
So, libvirt should add support for assigning pci devices in non-0 domains (and it's reasonably simple to do so), but it will only work on newer versions of qemu than what you have, so you would have to build your own (unsupported) upstream versions of both qemu and libvirt.
Do we have the support built into version 1.4.0 of qemu?
Yes, I believe so. (purely based on what I was told by a qemu developer, not personal experience).
Also, do I have to make the changes to the libvirt source code or is the support already available in some version, which I could use? Let me know, which versions or which source code will be a get place to start, if I want to add the support myself.
There is no support for it currently; it's one of those things that until now nobody has needed, so there was no push to do it. It shouldn't be too hard to add - we just need a capability flag saying that domains are supported in host-side PCI addresses, then add a %.4x and a def->source.subsys.u.pci.addr.domain to a printf (if the capability bit is set). Any new development should be done first on the head of git master. You can learn how to check that out from libvirt.org.
participants (2)
-
Charles Peri -
Laine Stump