6:45 a.m.
Hi,
I was testing Host device assignment in OpenStack environment where the driver name is
vfio or kvm.
My setup is as follow:
1. Fedora 21
2. Libvirt 1.3.0 which I compiled
3. OpenStack master
I have also other setups with older Libvirt version and the same OpenStack environment.
I notice that on my fedora environment the driver name is vfio were in my old environment
the driver name is kvm.
According to Libvirt documentation default is "vfio" on systems where the VFIO
driver is available and loaded, see [1]
I remove the vfio modules by removing vfio, vfio_iommu_type1, vfio_pci but when I boot a
vm the drive name is vfio
How can change the driver name to be kvm?
Another thing that I encounter is an error when suspending VM (in OpenStack environment)
when the driver name is vfio.
In such case I am getting the following error from Libvirt:
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher if ret == -1: raise
libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher libvirtError: internal
error: unable to execute QEMU command 'device_add': Device initialization failed.
I would appreciate for some pointers on what can cause this issue.
[1] https://libvirt.org/formatdomain.html#elementsHostDev
1:24 p.m.
New subject: [libvirt] Host device assignment driver name vfio/ kvm
On 03/29/2016 07:45 AM, Moshe Levi wrote:
Hi,
I was testing Host device assignment in OpenStack environment where
the driver name is vfio or kvm.
My setup is as follow:
1.Fedora 21
2.Libvirt 1.3.0 which I compiled
3.OpenStack master
I have also other setups with older Libvirt version and the same
OpenStack environment.
I notice that on my fedora environment the driver name is vfio were in
my old environment the driver name is kvm.
According to Libvirt documentation default is "vfio" on systems where
the VFIO driver is available and loaded, see [1]
I remove the vfio modules by removing vfio, vfio_iommu_type1, vfio_pci
but when I boot a vm the drive name is vfio
How can change the driver name to be kvm?
libvirt tries very hard to use vfio rather than legacy kvm, because
legacy kvm is old, deprecated, and "declared bad" :-). But it won't
changed it to vfio if you've explicitly said that you want to use kvm.
If you really want to use legacy kvm device assignment, manually set
that in the config. When you do that, if the system you're running on
doesn't support it, it will error out rather than switching.
Another thing that I encounter is an error when suspending VM (in
OpenStack environment) when the driver name is vfio.
In such case I am getting the following error from Libvirt:
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in
attachDeviceFlags
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher
if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags()
failed', dom=self)
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher
libvirtError: internal error: unable to execute QEMU command
'device_add': Device initialization failed.
I would appreciate for some pointers on what can cause this issue.
Assuming that openstack uses libvirt's virDomainSave API I would expect
suspending a guest to fail if it had an assigned device (since libvirt
implements this by "migrating to disk", and qemu doesn't allow migration
of a guest with an assigned device. But your problem is that it's trying
to *attach* a device, which I wouldn't consider to be a part of a save
or suspend or whatever operation. Is it possible to get more information
about what leads up to this?
[1] https://libvirt.org/formatdomain.html#elementsHostDev
<https://libvirt.org/formatdomain.html#elementsHostDev>
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
10:58 a.m.
New subject: [libvirt] Host device assignment driver name vfio/ kvm
Thanks Laine,
Adding the driver_name o the config did the trick, thanks ☺
Regarding the vifo error it seem that openstack does roleback if operation failed so that
why you see the virDomainAttachDeviceFlags
Anyhow I found that the in qemu 2.1 suspend is not working (I got error [1] ) so I
upgrade to qemu 2.5 and then suspend work but it failed on resume.
So Just to clarify vfio is not supporting “attach device” right? is qemu going to support
it?
Now I wonder if I need to hardcode in the hostdev config to be with driver_name=kvm…
[1] -ERROR:qom/object.c:725:object_unref: assertion failed: (obj->ref > 0)
From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump
Sent: Wednesday, March 30, 2016 9:25 PM
To: Libvirt <libvir-list(a)redhat.com>
Cc: Moshe Levi <moshele(a)mellanox.com>
Subject: Re: [libvirt] Host device assignment driver name vfio/ kvm
On 03/29/2016 07:45 AM, Moshe Levi wrote:
Hi,
I was testing Host device assignment in OpenStack environment where the driver name is
vfio or kvm.
My setup is as follow:
1. Fedora 21
2. Libvirt 1.3.0 which I compiled
3. OpenStack master
I have also other setups with older Libvirt version and the same OpenStack environment.
I notice that on my fedora environment the driver name is vfio were in my old environment
the driver name is kvm.
According to Libvirt documentation default is "vfio" on systems where the VFIO
driver is available and loaded, see [1]
I remove the vfio modules by removing vfio, vfio_iommu_type1, vfio_pci but when I boot a
vm the drive name is vfio
How can change the driver name to be kvm?
libvirt tries very hard to use vfio rather than legacy kvm, because legacy kvm is old,
deprecated, and "declared bad" :-). But it won't changed it to vfio if
you've explicitly said that you want to use kvm. If you really want to use legacy kvm
device assignment, manually set that in the config. When you do that, if the system
you're running on doesn't support it, it will error out rather than switching.
Another thing that I encounter is an error when suspending VM (in OpenStack environment)
when the driver name is vfio.
In such case I am getting the following error from Libvirt:
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in
attachDeviceFlags
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher if ret == -1: raise
libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher libvirtError: internal
error: unable to execute QEMU command 'device_add': Device initialization
failed.
I would appreciate for some pointers on what can cause this issue.
Assuming that openstack uses libvirt's virDomainSave API I would expect suspending a
guest to fail if it had an assigned device (since libvirt implements this by
"migrating to disk", and qemu doesn't allow migration of a guest with an
assigned device. But your problem is that it's trying to *attach* a device, which I
wouldn't consider to be a part of a save or suspend or whatever operation. Is it
possible to get more information about what leads up to this?
[1] https://libvirt.org/formatdomain.html#elementsHostDev
--
libvir-list mailing list
libvir-list@redhat.com<mailto:libvir-list@redhat.com>
https://www.redhat.com/mailman/listinfo/libvir-list
11:07 a.m.
New subject: [libvirt] Host device assignment driver name vfio/ kvm
On 03/31/2016 11:58 AM, Moshe Levi wrote:
Thanks Laine,
Adding the driver_name o the config did the trick, thanks J
Regarding the vifo error it seem that openstack does roleback if
operation failed so that why you see the virDomainAttachDeviceFlags
Anyhow I found that the in qemu 2.1 suspend is not working (I got
error [1] ) so I upgrade to qemu 2.5 and then suspend work but it
failed on resume.
So Just to clarify vfio is not supporting “attach device” right? is
qemu going to support it?
Not at all. attach device works just fine with vfio (that's how hotplug
is done). I've just been using it myself in some testing. (note that
vfio has been supported since kernel 3.6 and libvirt 1.0.5, i.e. a "very
long time")
Now I wonder if I need to hardcode in the hostdev config to be with
driver_name=kvm…
I certainly hope not. You would be viciously attacked by the keeper of
vfio for your transgression! :-)
Seriously, there should be no reason whatsoever to force driver
name='kvm'. This was made configurable *only* as a fallback in case
errors were encountered with vfio during its early days. As a matter of
fact, I'm pretty sure that RHEL7 has legacy kvm device assignment
completely disabled. If you are needing to force legacy kvm device
assignment to make your setup work, then there is a bug somewhere that
needs to be investigated and squashed.
[1] -ERROR:qom/object.c:725:object_unref: assertion failed:
(obj->ref > 0)
*From:*sendmail [mailto:justsendmailnothingelse@gmail.com] *On Behalf
Of *Laine Stump
*Sent:* Wednesday, March 30, 2016 9:25 PM
*To:* Libvirt <libvir-list(a)redhat.com>
*Cc:* Moshe Levi <moshele(a)mellanox.com>
*Subject:* Re: [libvirt] Host device assignment driver name vfio/ kvm
On 03/29/2016 07:45 AM, Moshe Levi wrote:
Hi,
I was testing Host device assignment in OpenStack environment
where the driver name is vfio or kvm.
My setup is as follow:
1.Fedora 21
2.Libvirt 1.3.0 which I compiled
3.OpenStack master
I have also other setups with older Libvirt version and the same
OpenStack environment.
I notice that on my fedora environment the driver name is vfio
were in my old environment the driver name is kvm.
According to Libvirt documentation default is "vfio" on systems
where the VFIO driver is available and loaded, see [1]
I remove the vfio modules by removing vfio, vfio_iommu_type1,
vfio_pci but when I boot a vm the drive name is vfio
How can change the driver name to be kvm?
libvirt tries very hard to use vfio rather than legacy kvm, because
legacy kvm is old, deprecated, and "declared bad" :-). But it won't
changed it to vfio if you've explicitly said that you want to use kvm.
If you really want to use legacy kvm device assignment, manually set
that in the config. When you do that, if the system you're running on
doesn't support it, it will error out rather than switching.
Another thing that I encounter is an error when suspending VM (in
OpenStack environment) when the driver name is vfio.
In such case I am getting the following error from Libvirt:
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in
attachDeviceFlags
2016-03-28 11:42:59.527 1966 ERROR
oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError
('virDomainAttachDeviceFlags() failed', dom=self)
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher
libvirtError: internal error: unable to execute QEMU command
'device_add': Device initialization failed.
I would appreciate for some pointers on what can cause this issue.
Assuming that openstack uses libvirt's virDomainSave API I would
expect suspending a guest to fail if it had an assigned device (since
libvirt implements this by "migrating to disk", and qemu doesn't allow
migration of a guest with an assigned device. But your problem is that
it's trying to *attach* a device, which I wouldn't consider to be a
part of a save or suspend or whatever operation. Is it possible to get
more information about what leads up to this?
[1] https://libvirt.org/formatdomain.html#elementsHostDev
<https://libvirt.org/formatdomain.html#elementsHostDev>
--
libvir-list mailing list
libvir-list(a)redhat.com <mailto:libvir-list@redhat.com>
https://www.redhat.com/mailman/listinfo/libvir-list
4 a.m.
New subject: [libvirt] Host device assignment driver name vfio/ kvm
Hi,
Ok, so here are the errors I get when using vfio/( kvm is working fine)
This is the Error [1] I am getting on resume when using vfio. And [2] is the ls output on
the /dev/vfio/
Please note that I am trying to attach pci which is VF
My setup is:
Fedora 21
Libvirt 1.3.0
Qemu 2.5
Kernel 3.17.4-301.fc21.x86_64
OpenStack – master.
Let me know if you need more information.
[1] –
root@r-dcs78:/var/log/libvirt/qemu# cat ./instance-0000000e.log
2016-03-31 15:31:09.895+0000: starting up libvirt version: 1.3.0, package: 1.fc21
(Unknown, 2016-02-25-10:56:09, r-ufm231.mtr.labs.mlnx), qemu version: 2.5.0
(qemu-2.5.0-11.fc21)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none
/usr/bin/qemu-kvm -name instance-0000000e -S -machine pc-i440fx-2.5,accel=kvm,usb=off -m
2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid
a809a3b5-ba75-41e6-889f-ffeebecfe54e -smbios 'type=1,manufacturer=OpenStack
Foundation,product=OpenStack
Nova,version=13.0.0,serial=533a28c3-7519-4e6d-8cdb-4b1f72649e71,uuid=a809a3b5-ba75-41e6-889f-ffeebecfe54e,family=Virtual
Machine' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-0000000e/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global
kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none
-device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -chardev
file,id=charserial0,path=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device
isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
vfio-pci,host=04:00.7,id=hostdev0,bus=pci.0,addr=0x4 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/29 (label charserial1)
2016-03-31T15:31:46.775166Z qemu-system-x86_64: terminating on signal 15 from pid 6866
2016-03-31 15:32:07.547+0000: starting up libvirt version: 1.3.0, package: 1.fc21
(Unknown, 2016-02-25-10:56:09, r-ufm231.mtr.labs.mlnx), qemu version: 2.5.0
(qemu-2.5.0-11.fc21)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none
/usr/bin/qemu-kvm -name instance-0000000e -S -machine pc-i440fx-2.5,accel=kvm,usb=off -m
2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid
a809a3b5-ba75-41e6-889f-ffeebecfe54e -smbios 'type=1,manufacturer=OpenStack
Foundation,product=OpenStack
Nova,version=13.0.0,serial=533a28c3-7519-4e6d-8cdb-4b1f72649e71,uuid=a809a3b5-ba75-41e6-889f-ffeebecfe54e,family=Virtual
Machine' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-0000000e/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global
kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none
-device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -chardev
file,id=charserial0,path=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device
isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -incoming defer -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/29 (label charserial1)
vfio: error opening /dev/vfio/57: Permission denied
vfio: failed to get group 57
2016-03-31T16:05:54.975312Z qemu-system-x86_64: terminating on signal 15 from pid 6866
[2] root@r-dcs78:/var/log/libvirt/qemu# ls -l /dev/vfio/
total 0
crw-rw-rw- 1 root root 10, 196 Mar 29 11:11 vfio
From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump
Sent: Thursday, March 31, 2016 7:07 PM
To: Libvirt <libvir-list(a)redhat.com>
Cc: Moshe Levi <moshele(a)mellanox.com>
Subject: Re: [libvirt] Host device assignment driver name vfio/ kvm
On 03/31/2016 11:58 AM, Moshe Levi wrote:
Thanks Laine,
Adding the driver_name o the config did the trick, thanks ☺
Regarding the vifo error it seem that openstack does roleback if operation failed so that
why you see the virDomainAttachDeviceFlags
Anyhow I found that the in qemu 2.1 suspend is not working (I got error [1] ) so I
upgrade to qemu 2.5 and then suspend work but it failed on resume.
So Just to clarify vfio is not supporting “attach device” right? is qemu going to support
it?
Not at all. attach device works just fine with vfio (that's how hotplug is done).
I've just been using it myself in some testing. (note that vfio has been supported
since kernel 3.6 and libvirt 1.0.5, i.e. a "very long time")
Now I wonder if I need to hardcode in the hostdev config to be with driver_name=kvm…
I certainly hope not. You would be viciously attacked by the keeper of vfio for your
transgression! :-)
Seriously, there should be no reason whatsoever to force driver name='kvm'. This
was made configurable *only* as a fallback in case errors were encountered with vfio
during its early days. As a matter of fact, I'm pretty sure that RHEL7 has legacy kvm
device assignment completely disabled. If you are needing to force legacy kvm device
assignment to make your setup work, then there is a bug somewhere that needs to be
investigated and squashed.
[1] -ERROR:qom/object.c:725:object_unref: assertion failed: (obj->ref > 0)
From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump
Sent: Wednesday, March 30, 2016 9:25 PM
To: Libvirt <libvir-list@redhat.com><mailto:libvir-list@redhat.com>
Cc: Moshe Levi <moshele@mellanox.com><mailto:moshele@mellanox.com>
Subject: Re: [libvirt] Host device assignment driver name vfio/ kvm
On 03/29/2016 07:45 AM, Moshe Levi wrote:
Hi,
I was testing Host device assignment in OpenStack environment where the driver name is
vfio or kvm.
My setup is as follow:
1. Fedora 21
2. Libvirt 1.3.0 which I compiled
3. OpenStack master
I have also other setups with older Libvirt version and the same OpenStack environment.
I notice that on my fedora environment the driver name is vfio were in my old environment
the driver name is kvm.
According to Libvirt documentation default is "vfio" on systems where the VFIO
driver is available and loaded, see [1]
I remove the vfio modules by removing vfio, vfio_iommu_type1, vfio_pci but when I boot a
vm the drive name is vfio
How can change the driver name to be kvm?
libvirt tries very hard to use vfio rather than legacy kvm, because legacy kvm is old,
deprecated, and "declared bad" :-). But it won't changed it to vfio if
you've explicitly said that you want to use kvm. If you really want to use legacy kvm
device assignment, manually set that in the config. When you do that, if the system
you're running on doesn't support it, it will error out rather than switching.
Another thing that I encounter is an error when suspending VM (in OpenStack environment)
when the driver name is vfio.
In such case I am getting the following error from Libvirt:
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in
attachDeviceFlags
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher if ret == -1: raise
libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher libvirtError: internal
error: unable to execute QEMU command 'device_add': Device initialization
failed.
I would appreciate for some pointers on what can cause this issue.
Assuming that openstack uses libvirt's virDomainSave API I would expect suspending a
guest to fail if it had an assigned device (since libvirt implements this by
"migrating to disk", and qemu doesn't allow migration of a guest with an
assigned device. But your problem is that it's trying to *attach* a device, which I
wouldn't consider to be a part of a save or suspend or whatever operation. Is it
possible to get more information about what leads up to this?
[1] https://libvirt.org/formatdomain.html#elementsHostDev
--
libvir-list mailing list
libvir-list@redhat.com<mailto:libvir-list@redhat.com>
https://www.redhat.com/mailman/listinfo/libvir-list
11:14 a.m.
New subject: [libvirt] Host device assignment driver name vfio/ kvm
(please configure your email client to properly quote earlier messages,
and reply inline rather as a top-post; it makes it easier for others to
determine the context of replies.)
On 04/03/2016 05:00 AM, Moshe Levi wrote:
Hi,
Ok, so here are the errors I get when using vfio/( kvm is working fine)
This is the Error [1] I am getting on resume when using vfio. And [2]
is the ls output on the /dev/vfio/
Please note that I am trying to attach pci which is VF
My setup is:
Fedora 21
Libvirt 1.3.0
Qemu 2.5
Kernel 3.17.4-301.fc21.x86_64
OpenStack – master.
Let me know if you need more information.
[1] –
root@r-dcs78:/var/log/libvirt/qemu# cat ./instance-0000000e.log
2016-03-31 15:31:09.895+0000: starting up libvirt version: 1.3.0,
package: 1.fc21 (Unknown, 2016-02-25-10:56:09,
r-ufm231.mtr.labs.mlnx), qemu version: 2.5.0 (qemu-2.5.0-11.fc21)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name instance-0000000e -S
-machine pc-i440fx-2.5,accel=kvm,usb=off -m 2048 -realtime mlock=off
-smp 1,sockets=1,cores=1,threads=1 -uuid
a809a3b5-ba75-41e6-889f-ffeebecfe54e -smbios
'type=1,manufacturer=OpenStack Foundation,product=OpenStack
Nova,version=13.0.0,serial=533a28c3-7519-4e6d-8cdb-4b1f72649e71,uuid=a809a3b5-ba75-41e6-889f-ffeebecfe54e,family=Virtual
Machine' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-0000000e/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none
-device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1
-chardev
file,id=charserial0,path=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev
pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1
-vnc 127.0.0.1:0 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
vfio-pci,host=04:00.7,id=hostdev0,bus=pci.0,addr=0x4 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/29 (label charserial1)
2016-03-31T15:31:46.775166Z qemu-system-x86_64: terminating on signal
15 from pid 6866
2016-03-31 15:32:07.547+0000: starting up libvirt version: 1.3.0,
package: 1.fc21 (Unknown, 2016-02-25-10:56:09,
r-ufm231.mtr.labs.mlnx), qemu version: 2.5.0 (qemu-2.5.0-11.fc21)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name instance-0000000e -S
-machine pc-i440fx-2.5,accel=kvm,usb=off -m 2048 -realtime mlock=off
-smp 1,sockets=1,cores=1,threads=1 -uuid
a809a3b5-ba75-41e6-889f-ffeebecfe54e -smbios
'type=1,manufacturer=OpenStack Foundation,product=OpenStack
Nova,version=13.0.0,serial=533a28c3-7519-4e6d-8cdb-4b1f72649e71,uuid=a809a3b5-ba75-41e6-889f-ffeebecfe54e,family=Virtual
Machine' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-0000000e/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none
-device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1
-chardev
file,id=charserial0,path=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev
pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1
-vnc 127.0.0.1:0 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -incoming defer -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/29 (label charserial1)
vfio: error opening /dev/vfio/57: Permission denied
vfio: failed to get group 57
It's very strange that this error follows a commandline that doesn't
have any "-device vfio-pci" in it, and the previous commandline (which
*did* have a "-device vfio-pci") had no error message. Is the device
being hotplugged? If not, why isn't it in the commandline? Can you
include the XML for the device in your next reply? (Since you're
assigning a network device, I'm hoping that you're using <interface
type='hostdev'> rather than <hostdev> - the former allows you to set the
VF's MAC address / vlan tag after detaching it from the host driver and
before assigning it to the guest)
Alex: what could be the reasons for permission-denied when opening the
node for the iommu group? Could this be an issue with other devices
being in the same iommu group? (I would hope this isn't the case, since
the device in question is apparently an SRIOV VF, so it should have its
own iommu group).
I'll be offline for about 6 days starting Monday AM, so may not be able
to participate in this discussion much more. Good luck!
2016-03-31T16:05:54.975312Z qemu-system-x86_64: terminating on signal
15 from pid 6866
[2] root@r-dcs78:/var/log/libvirt/qemu# ls -l /dev/vfio/
total 0
crw-rw-rw- 1 root root 10, 196 Mar 29 11:11 vfio
*From:*sendmail [mailto:justsendmailnothingelse@gmail.com] *On Behalf
Of *Laine Stump
*Sent:* Thursday, March 31, 2016 7:07 PM
*To:* Libvirt <libvir-list(a)redhat.com>
*Cc:* Moshe Levi <moshele(a)mellanox.com>
*Subject:* Re: [libvirt] Host device assignment driver name vfio/ kvm
On 03/31/2016 11:58 AM, Moshe Levi wrote:
Thanks Laine,
Adding the driver_name o the config did the trick, thanks J
Regarding the vifo error it seem that openstack does roleback if
operation failed so that why you see the virDomainAttachDeviceFlags
Anyhow I found that the in qemu 2.1 suspend is not working (I got
error [1] ) so I upgrade to qemu 2.5 and then suspend work but it
failed on resume.
So Just to clarify vfio is not supporting “attach device” right?
is qemu going to support it?
Not at all. attach device works just fine with vfio (that's how
hotplug is done). I've just been using it myself in some testing.
(note that vfio has been supported since kernel 3.6 and libvirt 1.0.5,
i.e. a "very long time")
Now I wonder if I need to hardcode in the hostdev config to be
with driver_name=kvm…
I certainly hope not. You would be viciously attacked by the keeper of
vfio for your transgression! :-)
Seriously, there should be no reason whatsoever to force driver
name='kvm'. This was made configurable *only* as a fallback in case
errors were encountered with vfio during its early days. As a matter
of fact, I'm pretty sure that RHEL7 has legacy kvm device assignment
completely disabled. If you are needing to force legacy kvm device
assignment to make your setup work, then there is a bug somewhere that
needs to be investigated and squashed.
[1] -ERROR:qom/object.c:725:object_unref: assertion failed:
(obj->ref > 0)
*From:*sendmail [mailto:justsendmailnothingelse@gmail.com] *On
Behalf Of *Laine Stump
*Sent:* Wednesday, March 30, 2016 9:25 PM
*To:* Libvirt <libvir-list(a)redhat.com> <mailto:libvir-list@redhat.com>
*Cc:* Moshe Levi <moshele(a)mellanox.com> <mailto:moshele@mellanox.com>
*Subject:* Re: [libvirt] Host device assignment driver name vfio/ kvm
On 03/29/2016 07:45 AM, Moshe Levi wrote:
Hi,
I was testing Host device assignment in OpenStack
environment where the driver name is vfio or kvm.
My setup is as follow:
1.Fedora 21
2.Libvirt 1.3.0 which I compiled
3.OpenStack master
I have also other setups with older Libvirt version and the
same OpenStack environment.
I notice that on my fedora environment the driver name is vfio
were in my old environment the driver name is kvm.
According to Libvirt documentation default is "vfio" on
systems where the VFIO driver is available and loaded, see [1]
I remove the vfio modules by removing vfio, vfio_iommu_type1,
vfio_pci but when I boot a vm the drive name is vfio
How can change the driver name to be kvm?
libvirt tries very hard to use vfio rather than legacy kvm,
because legacy kvm is old, deprecated, and "declared bad" :-). But
it won't changed it to vfio if you've explicitly said that you
want to use kvm. If you really want to use legacy kvm device
assignment, manually set that in the config. When you do that, if
the system you're running on doesn't support it, it will error out
rather than switching.
Another thing that I encounter is an error when suspending VM
(in OpenStack environment) when the driver name is vfio.
In such case I am getting the following error from Libvirt:
2016-03-28 11:42:59.527 1966 ERROR
oslo_messaging.rpc.dispatcher File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in
attachDeviceFlags
2016-03-28 11:42:59.527 1966 ERROR
oslo_messaging.rpc.dispatcher if ret == -1: raise
libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2016-03-28 11:42:59.527 1966 ERROR
oslo_messaging.rpc.dispatcher libvirtError: internal error:
unable to execute QEMU command 'device_add': Device
initialization failed.
I would appreciate for some pointers on what can cause this issue.
Assuming that openstack uses libvirt's virDomainSave API I would
expect suspending a guest to fail if it had an assigned device
(since libvirt implements this by "migrating to disk", and qemu
doesn't allow migration of a guest with an assigned device. But
your problem is that it's trying to *attach* a device, which I
wouldn't consider to be a part of a save or suspend or whatever
operation. Is it possible to get more information about what leads
up to this?
[1] https://libvirt.org/formatdomain.html#elementsHostDev
--
libvir-list mailing list
libvir-list(a)redhat.com <mailto:libvir-list@redhat.com>
https://www.redhat.com/mailman/listinfo/libvir-list
10:54 a.m.
New subject: [libvirt] Host device assignment driver name vfio/ kvm
From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump
Sent: Sunday, April 03, 2016 7:14 PM
To: Libvirt <libvir-list(a)redhat.com>
Cc: Moshe Levi <moshele(a)mellanox.com>; Alex Williamson
<Alex.Williamson(a)redhat.com>
Subject: Re: [libvirt] Host device assignment driver name vfio/ kvm
(please configure your email client to properly quote earlier messages, and reply inline
rather as a top-post; it makes it easier for others to determine the context of
replies.)
On 04/03/2016 05:00 AM, Moshe Levi wrote:
Hi,
Ok, so here are the errors I get when using vfio/( kvm is working fine)
This is the Error [1] I am getting on resume when using vfio. And [2] is the ls output on
the /dev/vfio/
Please note that I am trying to attach pci which is VF
My setup is:
Fedora 21
Libvirt 1.3.0
Qemu 2.5
Kernel 3.17.4-301.fc21.x86_64
OpenStack – master.
Let me know if you need more information.
[1] –
root@r-dcs78:/var/log/libvirt/qemu# cat ./instance-0000000e.log
2016-03-31 15:31:09.895+0000: starting up libvirt version: 1.3.0, package: 1.fc21
(Unknown, 2016-02-25-10:56:09, r-ufm231.mtr.labs.mlnx), qemu version: 2.5.0
(qemu-2.5.0-11.fc21)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none
/usr/bin/qemu-kvm -name instance-0000000e -S -machine pc-i440fx-2.5,accel=kvm,usb=off -m
2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid
a809a3b5-ba75-41e6-889f-ffeebecfe54e -smbios 'type=1,manufacturer=OpenStack
Foundation,product=OpenStack
Nova,version=13.0.0,serial=533a28c3-7519-4e6d-8cdb-4b1f72649e71,uuid=a809a3b5-ba75-41e6-889f-ffeebecfe54e,family=Virtual
Machine' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-0000000e/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global
kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none
-device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -chardev
file,id=charserial0,path=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device
isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
vfio-pci,host=04:00.7,id=hostdev0,bus=pci.0,addr=0x4 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/29 (label charserial1)
2016-03-31T15:31:46.775166Z qemu-system-x86_64: terminating on signal 15 from pid 6866
2016-03-31 15:32:07.547+0000: starting up libvirt version: 1.3.0, package: 1.fc21
(Unknown, 2016-02-25-10:56:09, r-ufm231.mtr.labs.mlnx), qemu version: 2.5.0
(qemu-2.5.0-11.fc21)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none
/usr/bin/qemu-kvm -name instance-0000000e -S -machine pc-i440fx-2.5,accel=kvm,usb=off -m
2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid
a809a3b5-ba75-41e6-889f-ffeebecfe54e -smbios 'type=1,manufacturer=OpenStack
Foundation,product=OpenStack
Nova,version=13.0.0,serial=533a28c3-7519-4e6d-8cdb-4b1f72649e71,uuid=a809a3b5-ba75-41e6-889f-ffeebecfe54e,family=Virtual
Machine' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-0000000e/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global
kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none
-device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -chardev
file,id=charserial0,path=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device
isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -incoming defer -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/29 (label charserial1)
vfio: error opening /dev/vfio/57: Permission denied
vfio: failed to get group 57
It's very strange that this error follows a commandline that doesn't have any
"-device vfio-pci" in it, and the previous commandline (which *did* have a
"-device vfio-pci") had no error message. Is the device being hotplugged? If
not, why isn't it in the commandline? Can you include the XML for the device in your
next reply? (Since you're assigning a network device, I'm hoping that you're
using <interface type='hostdev'> rather than <hostdev> - the former
allows you to set the VF's MAC address / vlan tag after detaching it from the host
driver and before assigning it to the guest)
I upgrade my OS to Fedora 23 with kernel-4.2.3-300.fc23.x86_64 and it working now, so it
maybe a kernel issue.
In my case I am using hostdev because the VF is InfiniBand Interface, but I had the same
problems with <interface type=’hostdev’> on other setup (I attached the vm xml to
the mail)
Alex: what could be the reasons for permission-denied when opening the node for the iommu
group? Could this be an issue with other devices being in the same iommu group? (I would
hope this isn't the case, since the device in question is apparently an SRIOV VF, so
it should have its own iommu group).
I'll be offline for about 6 days starting Monday AM, so may not be able to participate
in this discussion much more. Good luck!
2016-03-31T16:05:54.975312Z qemu-system-x86_64: terminating on signal 15 from pid 6866
[2] root@r-dcs78:/var/log/libvirt/qemu# ls -l /dev/vfio/
total 0
crw-rw-rw- 1 root root 10, 196 Mar 29 11:11 vfio
From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump
Sent: Thursday, March 31, 2016 7:07 PM
To: Libvirt <libvir-list@redhat.com><mailto:libvir-list@redhat.com>
Cc: Moshe Levi <moshele@mellanox.com><mailto:moshele@mellanox.com>
Subject: Re: [libvirt] Host device assignment driver name vfio/ kvm
On 03/31/2016 11:58 AM, Moshe Levi wrote:
Thanks Laine,
Adding the driver_name o the config did the trick, thanks ☺
Regarding the vifo error it seem that openstack does roleback if operation failed so that
why you see the virDomainAttachDeviceFlags
Anyhow I found that the in qemu 2.1 suspend is not working (I got error [1] ) so I
upgrade to qemu 2.5 and then suspend work but it failed on resume.
So Just to clarify vfio is not supporting “attach device” right? is qemu going to support
it?
Not at all. attach device works just fine with vfio (that's how hotplug is done).
I've just been using it myself in some testing. (note that vfio has been supported
since kernel 3.6 and libvirt 1.0.5, i.e. a "very long time")
Now I wonder if I need to hardcode in the hostdev config to be with driver_name=kvm…
I certainly hope not. You would be viciously attacked by the keeper of vfio for your
transgression! :-)
Seriously, there should be no reason whatsoever to force driver name='kvm'. This
was made configurable *only* as a fallback in case errors were encountered with vfio
during its early days. As a matter of fact, I'm pretty sure that RHEL7 has legacy kvm
device assignment completely disabled. If you are needing to force legacy kvm device
assignment to make your setup work, then there is a bug somewhere that needs to be
investigated and squashed.
[1] -ERROR:qom/object.c:725:object_unref: assertion failed: (obj->ref > 0)
From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump
Sent: Wednesday, March 30, 2016 9:25 PM
To: Libvirt <libvir-list@redhat.com><mailto:libvir-list@redhat.com>
Cc: Moshe Levi <moshele@mellanox.com><mailto:moshele@mellanox.com>
Subject: Re: [libvirt] Host device assignment driver name vfio/ kvm
On 03/29/2016 07:45 AM, Moshe Levi wrote:
Hi,
I was testing Host device assignment in OpenStack environment where the driver name is
vfio or kvm.
My setup is as follow:
1. Fedora 21
2. Libvirt 1.3.0 which I compiled
3. OpenStack master
I have also other setups with older Libvirt version and the same OpenStack environment.
I notice that on my fedora environment the driver name is vfio were in my old environment
the driver name is kvm.
According to Libvirt documentation default is "vfio" on systems where the VFIO
driver is available and loaded, see [1]
I remove the vfio modules by removing vfio, vfio_iommu_type1, vfio_pci but when I boot a
vm the drive name is vfio
How can change the driver name to be kvm?
libvirt tries very hard to use vfio rather than legacy kvm, because legacy kvm is old,
deprecated, and "declared bad" :-). But it won't changed it to vfio if
you've explicitly said that you want to use kvm. If you really want to use legacy kvm
device assignment, manually set that in the config. When you do that, if the system
you're running on doesn't support it, it will error out rather than switching.
Another thing that I encounter is an error when suspending VM (in OpenStack environment)
when the driver name is vfio.
In such case I am getting the following error from Libvirt:
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in
attachDeviceFlags
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher if ret == -1: raise
libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2016-03-28 11:42:59.527 1966 ERROR oslo_messaging.rpc.dispatcher libvirtError: internal
error: unable to execute QEMU command 'device_add': Device initialization
failed.
I would appreciate for some pointers on what can cause this issue.
Assuming that openstack uses libvirt's virDomainSave API I would expect suspending a
guest to fail if it had an assigned device (since libvirt implements this by
"migrating to disk", and qemu doesn't allow migration of a guest with an
assigned device. But your problem is that it's trying to *attach* a device, which I
wouldn't consider to be a part of a save or suspend or whatever operation. Is it
possible to get more information about what leads up to this?
[1] https://libvirt.org/formatdomain.html#elementsHostDev
--
libvir-list mailing list
libvir-list@redhat.com<mailto:libvir-list@redhat.com>
https://www.redhat.com/mailman/listinfo/libvir-list
3184
days inactive
3190
days old
6 comments
2 participants
participants (2)
-
Laine Stump -
Moshe Levi