[libvirt] Issues with qemu-nbd over AF_UNIX and virDomainAttachDevice
print nbdxml <disk type='network' device='disk'> <driver name="qemu" type="qcow2"/> <source protocol="nbd"> <host name="deepakcs-lx" port="10809" transport="unix" socket="/tmp/mysock2" /> </source> <target dev="vdc" bus="virtio" /> </disk> dom.attachDevice(nbdxml)
Hi List, Facing issues trying to use virDomainAttachDevice API of libvirt from python. Here is what I am trying to do 1) Setup qemu-nbd to export a qcow2 disk image over unix socket on my localhost qemu-nbd -t -k /tmp/mysock2 /home/dpkshetty/work/img/iscsi_disk The above blocks. and i am able to use qemu-system-x86_64 -drive file=nbd:unix:/tmp/mysock2 Just to test it...and it works. Since currently the disk is a blank disk.. I see "No bootable device" and QEMU stops.. whcih is expected So qemu-nbd is setup and working properly 2) libvir: QEMU error : operation failed: open disk image file failed Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 400, in attachDevice if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self) libvirt.libvirtError: operation failed: open disk image file failed and I see... in the VM's log.... inet_connect_opts: connect(ipv4,deepakcs-lx.local.lan,192.168.1.7,10809): Connection refused 3) I tried usign virsh attach-device.. and the same problem happens 4) firewall/iptables are not running. SELinux is permissive 5) I tried appending -p 1111 to the qemu-nbd cmdline and using port="1111" in the XML.. but that doesn't help, see the same error Questions/Observations : 1) What am i doing wrong here ? In the <host> tag, port attr is mandatory.. but looking at `netstat -nptl` I don't see any process listeing on 10809 or 1111 (when i gave -p 1111 ). So does this mean there is no way for libvirt to just directly open the socket and use it.. Its trying to connect to the port i give in the XML and since no one is listening there.. it fails.. hence the error seen in the VM logs.... Has anybody tryign usign qemu-nbd and attaching that disk as I am doign above ? Is it legal, if not, why ? 2) The VM/Domain is active during all of the above steps. 3) IIUC... for the above usecase... port attr of <host> should be optional... such that libvirt can directly do the equivalent of ` -drive file=nbd:unix:/tmp/mysock2` and add this as a NBD device to the runnign guest/Domain... but currently port and hostname are mandatory. For unix sockets... hostname will always be local...so i feel both hostname & port should be optional for the above usecase ? 4) I haven't yet tried doing the above when guest/Domain is Inactive. 5) I haven't yet tried doing the above usign nbd-server or qemu-nbd usign -c option (which i believe starts qemu-nbd in server mode, but need nbd.ko kernel module to be loaded). I am hoping using either of these might get it working... since we have a process listening on a port on the host... but I wanted to get this working using plain unix sockets.. lightweight and doesn't need nbd.ko Comments/Suggestions appreciated. thanx, deepak
Il 24/05/2013 06:59, Deepak C Shetty ha scritto:
Hi List, Facing issues trying to use virDomainAttachDevice API of libvirt from python.
Here is what I am trying to do
1) Setup qemu-nbd to export a qcow2 disk image over unix socket on my localhost qemu-nbd -t -k /tmp/mysock2 /home/dpkshetty/work/img/iscsi_disk
The above blocks. and i am able to use qemu-system-x86_64 -drive file=nbd:unix:/tmp/mysock2
Just to test it...and it works. Since currently the disk is a blank disk.. I see "No bootable device" and QEMU stops.. whcih is expected So qemu-nbd is setup and working properly
print nbdxml <disk type='network' device='disk'> <driver name="qemu" type="qcow2"/> <source protocol="nbd"> <host name="deepakcs-lx" port="10809" transport="unix" socket="/tmp/mysock2" /> </source> <target dev="vdc" bus="virtio" /> </disk> dom.attachDevice(nbdxml)
2) libvir: QEMU error : operation failed: open disk image file failed Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 400, in attachDevice if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self) libvirt.libvirtError: operation failed: open disk image file failed
and I see... in the VM's log....
inet_connect_opts: connect(ipv4,deepakcs-lx.local.lan,192.168.1.7,10809): Connection refused
What version of libvirt is this? It should work starting from 1.0.4. Paolo
On 05/24/2013 12:38 PM, Paolo Bonzini wrote:
Il 24/05/2013 06:59, Deepak C Shetty ha scritto: What version of libvirt is this? It should work starting from 1.0.4.
Ah, thats what I seem to have missed. I was using the distro provided libvirt.. which seems to be old (0.9)! I will compile latest libvirt and try. I still have a Q.. for qemu-nbd exporting over Unix socket.. what should i put for port attr of <host> tag ? I am assuming its ignored for Unix sockets ?
Paolo
Il 24/05/2013 10:01, Deepak C Shetty ha scritto:
Il 24/05/2013 06:59, Deepak C Shetty ha scritto: What version of libvirt is this? It should work starting from 1.0.4.
Ah, thats what I seem to have missed. I was using the distro provided libvirt.. which seems to be old (0.9)! I will compile latest libvirt and try.
I still have a Q.. for qemu-nbd exporting over Unix socket.. what should i put for port attr of <host> tag ? I am assuming its ignored for Unix sockets ?
It's not required anymore. In fact it was a bug that transport='unix' was accepted in the old libvirt, because it didn't know how to process it. Paolo
On 05/25/2013 02:04 AM, Paolo Bonzini wrote:
Il 24/05/2013 10:01, Deepak C Shetty ha scritto:
Il 24/05/2013 06:59, Deepak C Shetty ha scritto: What version of libvirt is this? It should work starting from 1.0.4. Ah, thats what I seem to have missed. I was using the distro provided libvirt.. which seems to be old (0.9)! I will compile latest libvirt and try.
I still have a Q.. for qemu-nbd exporting over Unix socket.. what should i put for port attr of <host> tag ? I am assuming its ignored for Unix sockets ? It's not required anymore. In fact it was a bug that transport='unix' was accepted in the old libvirt, because it didn't know how to process it.
Paolo
I tried with the latest git version of libvirt (1.0.5). I still get the same error for trying to attach nbd device, where as file based device works. ./daemon/libvirtd --version /home/dpkshetty/work/newlibvirt2/libvirt/daemon/.libs/lt-libvirtd (libvirt) 1.0.5 My qemu-nbd exports the disk as... qemu-nbd -t -k /tmp/mysock2 /home/dpkshetty/work/img/iscsi_disk 1) My libvirtd started as ./run ./daemon/libvirtd and in this console i see... error : qemuMonitorTextAddDrive:2691 : operation failed: open disk image file failed 2) My python example ( I tried with virsh (started as ./run ./tools/virsh...) - see same error)
print filexml <disk type='file' device='disk'> <driver name="qemu" type="qcow2"/> <source file='/home/dpkshetty/work/img/iscsi_disk'> </source> <target dev="vdc" bus="virtio" /> </disk> dom.attachDevice(filexml) 0 help(dom)
dom.detachDevice(filexml) 0 nbdxml='''<disk type='network'>\n <driver name="qemu" type="qcow2"/>\n <source protocol="nbd">\n <host name="deepakcs-lx" port="1111" transport="unix" socket="/tmp/mysock2" />\n </source>\n <target dev="vdc" bus="virtio" />\n</disk>''' print nbdxml <disk type='network'> <driver name="qemu" type="qcow2"/> <source protocol="nbd"> <host name="deepakcs-lx" port="1111" transport="unix" socket="/tmp/mysock2" /> </source> <target dev="vdc" bus="virtio" /> </disk> dom.attachDevice(nbdxml) libvirt: QEMU Driver error : operation failed: open disk image file failed Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/dpkshetty/work/newlibvirt2/libvirt/python/libvirt.py", line 419, in attachDevice if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self) libvirt.libvirtError: operation failed: open disk image file failed
3) In nbdxml, i tried removing port="1111".. it didn't give any error with regards to port beign mandatory.. which si good.. but the attach device still gives the same error as above Is the `nbdxml` I am using for attachign a qemu-nbd exported drive, correct ? thanx, deepak
Il 26/05/2013 15:54, Deepak C Shetty ha scritto:
3) In nbdxml, i tried removing port="1111".. it didn't give any error with regards to port beign mandatory.. which si good.. but the attach device still gives the same error as above
Is the `nbdxml` I am using for attachign a qemu-nbd exported drive, correct ?
What version of QEMU is this? Can you search the logs for the QMP commands related to the hotplug? Paolo
On 05/27/2013 02:00 PM, Paolo Bonzini wrote:
Il 26/05/2013 15:54, Deepak C Shetty ha scritto:
3) In nbdxml, i tried removing port="1111".. it didn't give any error with regards to port beign mandatory.. which si good.. but the attach device still gives the same error as above
Is the `nbdxml` I am using for attachign a qemu-nbd exported drive, correct ? What version of QEMU is this?
Can you search the logs for the QMP commands related to the hotplug?
Do you mean starting my domain using -d option for qemu, which dumps the log in /tmp/qemu.log ? I am using a VM started from virt-manager.... i don't see a way to pass -d to it from virt-manager... I can try using a hand-coded qemu cmdline tho' I assume when i am using python (import libvirt) and/or virsh.. it uses qemu-kvm qemu-kvm version is 1.0.1 Do i need to try with latest qemu git ? On irc.. you asked me to check if cold-plug works.. but attach-device is for active domain only... am i missing something here ? thanx, deepak
Il 27/05/2013 12:46, Deepak C Shetty ha scritto:
On 05/27/2013 02:00 PM, Paolo Bonzini wrote:
Il 26/05/2013 15:54, Deepak C Shetty ha scritto:
3) In nbdxml, i tried removing port="1111".. it didn't give any error with regards to port beign mandatory.. which si good.. but the attach device still gives the same error as above
Is the `nbdxml` I am using for attachign a qemu-nbd exported drive, correct ? What version of QEMU is this?
Can you search the logs for the QMP commands related to the hotplug?
Do you mean starting my domain using -d option for qemu, which dumps the log in /tmp/qemu.log ?
I mean libvirt.log. You can start libvirtd with these environment variables: LIBVIRT_DEBUG=debug LIBVIRT_LOG_OUTPUTS=1:file:/tmp/libvirt.log
I am using a VM started from virt-manager.... i don't see a way to pass -d to it from virt-manager... I can try using a hand-coded qemu cmdline tho'
I assume when i am using python (import libvirt) and/or virsh.. it uses qemu-kvm qemu-kvm version is 1.0.1
Ok.
Do i need to try with latest qemu git ?
No, I don't think so, but you can try (1.4.0 or 1.5.0 will do).
On irc.. you asked me to check if cold-plug works.. but attach-device is for active domain only... am i missing something here ?
You can try putting the <disk> item and start the VM. In this case the logs in /var/log/libvirt/qemu will be helpful, because they contain the command-line that is used to start QEMU. Paolo
On 05/27/2013 04:41 PM, Paolo Bonzini wrote:
Il 27/05/2013 12:46, Deepak C Shetty ha scritto:
On 05/27/2013 02:00 PM, Paolo Bonzini wrote:
Il 26/05/2013 15:54, Deepak C Shetty ha scritto:
3) In nbdxml, i tried removing port="1111".. it didn't give any error with regards to port beign mandatory.. which si good.. but the attach device still gives the same error as above
Is the `nbdxml` I am using for attachign a qemu-nbd exported drive, correct ? What version of QEMU is this?
Can you search the logs for the QMP commands related to the hotplug? Do you mean starting my domain using -d option for qemu, which dumps the log in /tmp/qemu.log ? I mean libvirt.log. You can start libvirtd with these environment variables:
LIBVIRT_DEBUG=debug LIBVIRT_LOG_OUTPUTS=1:file:/tmp/libvirt.log
dom.attachDevice(nbdxml)
Thanks, this helped. Here is the summary With latest libvirtd (1.0.5 built from git)... <disk type='network'> <driver name="qemu" type="qcow2"/> <source protocol="nbd"> <host name="deepakcs-lx" transport="unix" socket="/tmp/mysock2" /> </source> <target dev="vdc" bus="virtio" /> </disk> libvirt: QEMU Driver error : operation failed: open disk image file failed Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/dpkshetty/work/newlibvirt2/libvirt/python/libvirt.py", line 419, in attachDevice if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self) libvirt.libvirtError: operation failed: open disk image file failed * from /tmp/libvirt.log QEMU_MONITOR_IO_WRITE: mon=0x7f8308003d40 buf={"execute":"human-monitor-command","arguments":{"command-line":"drive_add dummy file=nbd:unix:/tmp/mysock2,if=none,id=drive-virtio-disk2,format=qcow2"},"id":"libvirt-390"} QEMU_MONITOR_RECV_REPLY: mon=0x7f8308003d40 reply={"return": "could not open disk image nbd:unix:/tmp/mysock2: Invalid argument\r\n", "id": "libvirt-390"} On command line... qemu-kvm -drive file=nbd:unix:/tmp/mysock2,format=qcow2 qemu-kvm: -drive file=nbd:unix:/tmp/mysock2,format=qcow2: could not open disk image nbd:unix:/tmp/mysock2: Invalid argument but qemu-kvm -drive file=nbd:unix:/tmp/mysock2 - works! So looks like format=qcow2 is causing issues!!! Then tried... (removing the <driver... altoghether) <disk type='network'> <source protocol="nbd"> <host name="deepakcs-lx" transport="unix" socket="/tmp/mysock2" /> </source> <target dev="vdc" bus="virtio" /> </disk>
dom.attachDevice(nbdxml) 0
dom.detachDevice(nbdxml) 0
* from /tmp/libvirt.log QEMU_MONITOR_IO_WRITE: mon=0x7f8308003d40 buf={"execute":"human-monitor-command","arguments":{"command-line":"drive_add dummy file=nbd:unix:/tmp/mysock2,if=none,id=drive-virtio-disk2"},"id":"libvirt-867"} This works and I was able to successfully add/remove a disk exported via qemu-nbd to a running VM !
I am using a VM started from virt-manager.... i don't see a way to pass -d to it from virt-manager... I can try using a hand-coded qemu cmdline tho'
I assume when i am using python (import libvirt) and/or virsh.. it uses qemu-kvm qemu-kvm version is 1.0.1 Ok.
Do i need to try with latest qemu git ? No, I don't think so, but you can try (1.4.0 or 1.5.0 will do).
I was using qemu-kvm version 1.0.1... for all of the ^^ above
On irc.. you asked me to check if cold-plug works.. but attach-device is for active domain only... am i missing something here ? You can try putting the <disk> item and start the VM. In this case the logs in /var/log/libvirt/qemu will be helpful, because they contain the command-line that is used to start QEMU.
Tried putting the above nbdxml usign virsh edit <domname> as an addnl disk and domain booted fine It throws the same error if u add format=qcow2 under <driver>... So looks like the right way to use NBD is *not* to specify format and let QEMU autosense it ? thanx, deepak
Paolo
Il 27/05/2013 15:41, Deepak C Shetty ha scritto:
Tried putting the above nbdxml usign virsh edit <domname> as an addnl disk and domain booted fine It throws the same error if u add format=qcow2 under <driver>...
So looks like the right way to use NBD is *not* to specify format and let QEMU autosense it ?
No, the format for NBD will in general be raw. qemu-nbd takes care of qcow2. This is because NBD lets you make the VM description agnostic of the format, which in general is desirable. Hence, raw is what you should specify most of the time; leaving format probing to QEMU will usually work but it is highly insecure. Older versions of qemu-nbd will always autosense the format; but because autosensing is insecure, newer versions (1.4.1 and newer) have a "-f" option to specify the format manually. In the newer versions that have "-f", the same option can be used if you want to specify the format option in the device XML; to do so, specify "-f raw" and add back the <driver> element to the XML. This however is generally not what you want to do. Paolo
participants (2)
-
Deepak C Shetty -
Paolo Bonzini