RBD volume not made available to Xen virtual guest on openSUSE 15.2 (with libvirt 6.0.0)
Hi there, Since upgrading to openSUSE 15.2 (which includes libvirt 6.0.0) the virtual guests don't get their RBD disks made available to them. On openSUSE 15.1 (which includes libvirt 5.1.0) that worked fine. The XML is as follows: <domain type='xen' id='7'> <name>mytwotel-a</name> <uuid>a56daa5d-c095-49d5-ae1b-00b38353614e</uuid> <description>mytwotel-a</description> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <cputune> <vcpupin vcpu='0' cpuset='8-23'/> </cputune> <os> <type arch='x86_64' machine='xenpv'>linux</type> <kernel>/usr/lib/grub2/x86_64-xen/grub.xen</kernel> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writethrough'/> <source protocol='rbd' name='guests/mytwotel-a'> <auth username='libvirt'> <secret type='ceph' uuid='3f88b59a-d85b-4b47-946d-a4c4cce3fec0'/> </auth> </source> <backingStore/> <target dev='xvda' bus='xen'/> </disk> <controller type='xenbus' index='0'/> <interface type='bridge'> <mac address='00:16:3e:a3:ba:9f'/> <source bridge='br0'/> <target dev='vif7.0'/> </interface> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='xen' port='0'/> </console> <input type='mouse' bus='xen'/> <input type='keyboard' bus='xen'/> <memballoon model='xen'/> </devices> </domain> The virtual guest starts, but then sits in the Grub 2 boot prompt because the disk is not available. The qemu log shows: qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory ... I tried to strace libvirtd. The results are as follows: On openSUSE 15.2 with libvirt 6.0.0 (not working), we see this: 1682 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f538aefd000 1682 mprotect(0x7f538aefd000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM IO], 8) = 0 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...> 1682 <... mmap resumed>) = 0x7f538adfc000 1682 mprotect(0x7f538adfc000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], <unfinished ...> 1682 <... rt_sigprocmask resumed>[BUS USR1 ALRM IO], 8) = 0 1682 write(2, "qemu-system-i386: failed to crea"..., 232 <unfinished ...> ... On the other hand, on openSUSE 15.1 with libvirt 5.1.0 (working), we see this: 16267 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory) 16267 stat("rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 0x7fff83e2e2b0) = -1 ENOENT (No such file or directory) 16267 access("/usr/lib64/qemu/block-rbd.so", F_OK) = 0 16267 stat("/usr/lib64/qemu/block-rbd.so", {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 openat(AT_FDCWD, "/usr/lib64/qemu/block-rbd.so", O_RDONLY|O_CLOEXEC) = 60 16267 read(60, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 &\0\0\0\0\0\0"..., 832) = 832 16267 fstat(60, {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 mmap(NULL, 2122672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 60, 0) = 0x7f8e6030f000 16267 mprotect(0x7f8e60315000, 2093056, PROT_NONE) = 0 16267 mmap(0x7f8e60514000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 60, 0x5000) = 0x7f8e60514000 16267 close(60) = 0 ... Note that the latter opens "/usr/lib64/qemu/block-rbd.so". That library *does* exist on openSUSE 15.2 but it doesn't seem to be used. I've tried to update libvirt to a newer version using the Open Build Service repos, but then ran into so many conflicting versions that I gave up. At this point I'm stuck. Does anyone have an idea I can try? Many thanks, Marcel
On a Friday in 2020, Marcel Juffermans wrote:
Hi there,
Since upgrading to openSUSE 15.2 (which includes libvirt 6.0.0) the virtual guests don't get their RBD disks made available to them. On openSUSE 15.1 (which includes libvirt 5.1.0) that worked fine. The XML is as follows:
[...]
I tried to strace libvirtd. The results are as follows:
On openSUSE 15.2 with libvirt 6.0.0 (not working), we see this:
1682 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f538aefd000 1682 mprotect(0x7f538aefd000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM IO], 8) = 0 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...> 1682 <... mmap resumed>) = 0x7f538adfc000 1682 mprotect(0x7f538adfc000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], <unfinished ...> 1682 <... rt_sigprocmask resumed>[BUS USR1 ALRM IO], 8) = 0 1682 write(2, "qemu-system-i386: failed to crea"..., 232 <unfinished ...> ...
On the other hand, on openSUSE 15.1 with libvirt 5.1.0 (working), we see this:
16267 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory) 16267 stat("rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 0x7fff83e2e2b0) = -1 ENOENT (No such file or directory) 16267 access("/usr/lib64/qemu/block-rbd.so", F_OK) = 0 16267 stat("/usr/lib64/qemu/block-rbd.so", {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 openat(AT_FDCWD, "/usr/lib64/qemu/block-rbd.so", O_RDONLY|O_CLOEXEC) = 60 16267 read(60, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 &\0\0\0\0\0\0"..., 832) = 832 16267 fstat(60, {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 mmap(NULL, 2122672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 60, 0) = 0x7f8e6030f000 16267 mprotect(0x7f8e60315000, 2093056, PROT_NONE) = 0 16267 mmap(0x7f8e60514000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 60, 0x5000) = 0x7f8e60514000 16267 close(60) = 0 ...
Note that the latter opens "/usr/lib64/qemu/block-rbd.so". That library *does* exist on openSUSE 15.2 but it doesn't seem to be used.
The error message comes from QEMU, what QEMU version are you using? Jano
I've tried to update libvirt to a newer version using the Open Build Service repos, but then ran into so many conflicting versions that I gave up.
At this point I'm stuck. Does anyone have an idea I can try?
Many thanks,
Marcel
It's QEMU 4.2.1-lp152.9.6.1. I've tried updating it from the Open Build Service repos but there's too many version conflicts. Marcel On 26/10/20 9:02 pm, Ján Tomko wrote:
On a Friday in 2020, Marcel Juffermans wrote:
Hi there,
Since upgrading to openSUSE 15.2 (which includes libvirt 6.0.0) the virtual guests don't get their RBD disks made available to them. On openSUSE 15.1 (which includes libvirt 5.1.0) that worked fine. The XML is as follows:
[...]
I tried to strace libvirtd. The results are as follows:
On openSUSE 15.2 with libvirt 6.0.0 (not working), we see this:
1682 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f538aefd000 1682 mprotect(0x7f538aefd000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM IO], 8) = 0 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...> 1682 <... mmap resumed>) = 0x7f538adfc000 1682 mprotect(0x7f538adfc000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], <unfinished ...> 1682 <... rt_sigprocmask resumed>[BUS USR1 ALRM IO], 8) = 0 1682 write(2, "qemu-system-i386: failed to crea"..., 232 <unfinished ...> ...
On the other hand, on openSUSE 15.1 with libvirt 5.1.0 (working), we see this:
16267 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory) 16267 stat("rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 0x7fff83e2e2b0) = -1 ENOENT (No such file or directory) 16267 access("/usr/lib64/qemu/block-rbd.so", F_OK) = 0 16267 stat("/usr/lib64/qemu/block-rbd.so", {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 openat(AT_FDCWD, "/usr/lib64/qemu/block-rbd.so", O_RDONLY|O_CLOEXEC) = 60 16267 read(60, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 &\0\0\0\0\0\0"..., 832) = 832 16267 fstat(60, {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 mmap(NULL, 2122672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 60, 0) = 0x7f8e6030f000 16267 mprotect(0x7f8e60315000, 2093056, PROT_NONE) = 0 16267 mmap(0x7f8e60514000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 60, 0x5000) = 0x7f8e60514000 16267 close(60) = 0 ...
Note that the latter opens "/usr/lib64/qemu/block-rbd.so". That library *does* exist on openSUSE 15.2 but it doesn't seem to be used.
The error message comes from QEMU, what QEMU version are you using?
Jano
I've tried to update libvirt to a newer version using the Open Build Service repos, but then ran into so many conflicting versions that I gave up.
At this point I'm stuck. Does anyone have an idea I can try?
Many thanks,
Marcel
Further to this, the error may be coming from QEMU, but the guests work fine when started using KVM (+ libvirt + QEMU) instead of Xen (+ libvirt + QEMU). So is it possible the problem is in libvirt, or rather, in the interaction between libvirt, Xen and QEMU? Marcel On 27/10/20 9:18 am, Marcel Juffermans wrote:
It's QEMU 4.2.1-lp152.9.6.1.
I've tried updating it from the Open Build Service repos but there's too many version conflicts.
Marcel On 26/10/20 9:02 pm, Ján Tomko wrote:
On a Friday in 2020, Marcel Juffermans wrote:
Hi there,
Since upgrading to openSUSE 15.2 (which includes libvirt 6.0.0) the virtual guests don't get their RBD disks made available to them. On openSUSE 15.1 (which includes libvirt 5.1.0) that worked fine. The XML is as follows:
[...]
I tried to strace libvirtd. The results are as follows:
On openSUSE 15.2 with libvirt 6.0.0 (not working), we see this:
1682 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f538aefd000 1682 mprotect(0x7f538aefd000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM IO], 8) = 0 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...> 1682 <... mmap resumed>) = 0x7f538adfc000 1682 mprotect(0x7f538adfc000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], <unfinished ...> 1682 <... rt_sigprocmask resumed>[BUS USR1 ALRM IO], 8) = 0 1682 write(2, "qemu-system-i386: failed to crea"..., 232 <unfinished ...> ...
On the other hand, on openSUSE 15.1 with libvirt 5.1.0 (working), we see this:
16267 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory) 16267 stat("rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 0x7fff83e2e2b0) = -1 ENOENT (No such file or directory) 16267 access("/usr/lib64/qemu/block-rbd.so", F_OK) = 0 16267 stat("/usr/lib64/qemu/block-rbd.so", {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 openat(AT_FDCWD, "/usr/lib64/qemu/block-rbd.so", O_RDONLY|O_CLOEXEC) = 60 16267 read(60, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 &\0\0\0\0\0\0"..., 832) = 832 16267 fstat(60, {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 mmap(NULL, 2122672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 60, 0) = 0x7f8e6030f000 16267 mprotect(0x7f8e60315000, 2093056, PROT_NONE) = 0 16267 mmap(0x7f8e60514000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 60, 0x5000) = 0x7f8e60514000 16267 close(60) = 0 ...
Note that the latter opens "/usr/lib64/qemu/block-rbd.so". That library *does* exist on openSUSE 15.2 but it doesn't seem to be used.
The error message comes from QEMU, what QEMU version are you using?
Jano
I've tried to update libvirt to a newer version using the Open Build Service repos, but then ran into so many conflicting versions that I gave up.
At this point I'm stuck. Does anyone have an idea I can try?
Many thanks,
Marcel
On 10/22/20 6:48 PM, Marcel Juffermans wrote:
Hi there,
Since upgrading to openSUSE 15.2 (which includes libvirt 6.0.0) the virtual guests don't get their RBD disks made available to them. On openSUSE 15.1 (which includes libvirt 5.1.0) that worked fine. The XML is as follows:
<domain type='xen' id='7'> <name>mytwotel-a</name> <uuid>a56daa5d-c095-49d5-ae1b-00b38353614e</uuid> <description>mytwotel-a</description> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <cputune> <vcpupin vcpu='0' cpuset='8-23'/> </cputune> <os> <type arch='x86_64' machine='xenpv'>linux</type> <kernel>/usr/lib/grub2/x86_64-xen/grub.xen</kernel> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writethrough'/> <source protocol='rbd' name='guests/mytwotel-a'> <auth username='libvirt'> <secret type='ceph' uuid='3f88b59a-d85b-4b47-946d-a4c4cce3fec0'/> </auth> </source> <backingStore/> <target dev='xvda' bus='xen'/> </disk> <controller type='xenbus' index='0'/> <interface type='bridge'> <mac address='00:16:3e:a3:ba:9f'/> <source bridge='br0'/> <target dev='vif7.0'/> </interface> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='xen' port='0'/> </console> <input type='mouse' bus='xen'/> <input type='keyboard' bus='xen'/> <memballoon model='xen'/> </devices> </domain>
The virtual guest starts, but then sits in the Grub 2 boot prompt because the disk is not available. The qemu log shows:
qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory ...
I tried to strace libvirtd. The results are as follows:
On openSUSE 15.2 with libvirt 6.0.0 (not working), we see this:
1682 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f538aefd000 1682 mprotect(0x7f538aefd000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM IO], 8) = 0 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...> 1682 <... mmap resumed>) = 0x7f538adfc000 1682 mprotect(0x7f538adfc000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], <unfinished ...> 1682 <... rt_sigprocmask resumed>[BUS USR1 ALRM IO], 8) = 0 1682 write(2, "qemu-system-i386: failed to crea"..., 232 <unfinished ...> ...
On the other hand, on openSUSE 15.1 with libvirt 5.1.0 (working), we see this:
16267 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory) 16267 stat("rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 0x7fff83e2e2b0) = -1 ENOENT (No such file or directory) 16267 access("/usr/lib64/qemu/block-rbd.so", F_OK) = 0 16267 stat("/usr/lib64/qemu/block-rbd.so", {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 openat(AT_FDCWD, "/usr/lib64/qemu/block-rbd.so", O_RDONLY|O_CLOEXEC) = 60 16267 read(60, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 &\0\0\0\0\0\0"..., 832) = 832 16267 fstat(60, {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 mmap(NULL, 2122672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 60, 0) = 0x7f8e6030f000 16267 mprotect(0x7f8e60315000, 2093056, PROT_NONE) = 0 16267 mmap(0x7f8e60514000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 60, 0x5000) = 0x7f8e60514000 16267 close(60) = 0 ...
Note that the latter opens "/usr/lib64/qemu/block-rbd.so". That library *does* exist on openSUSE 15.2 but it doesn't seem to be used.
It smells like a bug in the Leap 15.2 qemu package. Or perhaps in xen's libxl library, which controls the qemu processes used by VMs. I'd suggest entering a bug at bugzilla.opensuse.org.
I've tried to update libvirt to a newer version using the Open Build Service repos, but then ran into so many conflicting versions that I gave up.
At this point I'm stuck. Does anyone have an idea I can try?
We might be able to eliminate libxl by comparing the interaction with qemu between the working and non-working setups. E.g. enable debug in /etc/libvirt/libvirtd.conf, restart libvirtd, start the offending VM, then look in /var/log/libvirt/libxl/libxl-driver.log for the qemu command line created by libxl and qmp commands sent to qemu. If libxl prepares qemu similarly between the two, then the issue is likely in the qemu package. Regards, Jim
Thanks Jim, Looking at the logs for the working and non-working setups, the command line for QEMU is identical and the qmp commands almost the same: they both do "query-chardev" and "query-vnc" and the working setup does an additional "qmp_capabilities" which is likely not relevant. So I guess it must be in QEMU - I'll head over to the bug tracker. Thanks for your help! Marcel On 28/10/20 9:26 am, Jim Fehlig wrote:
On 10/22/20 6:48 PM, Marcel Juffermans wrote:
Hi there,
Since upgrading to openSUSE 15.2 (which includes libvirt 6.0.0) the virtual guests don't get their RBD disks made available to them. On openSUSE 15.1 (which includes libvirt 5.1.0) that worked fine. The XML is as follows:
<domain type='xen' id='7'> <name>mytwotel-a</name> <uuid>a56daa5d-c095-49d5-ae1b-00b38353614e</uuid> <description>mytwotel-a</description> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <cputune> <vcpupin vcpu='0' cpuset='8-23'/> </cputune> <os> <type arch='x86_64' machine='xenpv'>linux</type> <kernel>/usr/lib/grub2/x86_64-xen/grub.xen</kernel> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writethrough'/> <source protocol='rbd' name='guests/mytwotel-a'> <auth username='libvirt'> <secret type='ceph' uuid='3f88b59a-d85b-4b47-946d-a4c4cce3fec0'/> </auth> </source> <backingStore/> <target dev='xvda' bus='xen'/> </disk> <controller type='xenbus' index='0'/> <interface type='bridge'> <mac address='00:16:3e:a3:ba:9f'/> <source bridge='br0'/> <target dev='vif7.0'/> </interface> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='xen' port='0'/> </console> <input type='mouse' bus='xen'/> <input type='keyboard' bus='xen'/> <memballoon model='xen'/> </devices> </domain>
The virtual guest starts, but then sits in the Grub 2 boot prompt because the disk is not available. The qemu log shows:
qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory ...
I tried to strace libvirtd. The results are as follows:
On openSUSE 15.2 with libvirt 6.0.0 (not working), we see this:
1682 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f538aefd000 1682 mprotect(0x7f538aefd000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM IO], 8) = 0 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...> 1682 <... mmap resumed>) = 0x7f538adfc000 1682 mprotect(0x7f538adfc000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], <unfinished ...> 1682 <... rt_sigprocmask resumed>[BUS USR1 ALRM IO], 8) = 0 1682 write(2, "qemu-system-i386: failed to crea"..., 232 <unfinished ...> ...
On the other hand, on openSUSE 15.1 with libvirt 5.1.0 (working), we see this:
16267 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory) 16267 stat("rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 0x7fff83e2e2b0) = -1 ENOENT (No such file or directory) 16267 access("/usr/lib64/qemu/block-rbd.so", F_OK) = 0 16267 stat("/usr/lib64/qemu/block-rbd.so", {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 openat(AT_FDCWD, "/usr/lib64/qemu/block-rbd.so", O_RDONLY|O_CLOEXEC) = 60 16267 read(60, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 &\0\0\0\0\0\0"..., 832) = 832 16267 fstat(60, {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 mmap(NULL, 2122672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 60, 0) = 0x7f8e6030f000 16267 mprotect(0x7f8e60315000, 2093056, PROT_NONE) = 0 16267 mmap(0x7f8e60514000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 60, 0x5000) = 0x7f8e60514000 16267 close(60) = 0 ...
Note that the latter opens "/usr/lib64/qemu/block-rbd.so". That library *does* exist on openSUSE 15.2 but it doesn't seem to be used.
It smells like a bug in the Leap 15.2 qemu package. Or perhaps in xen's libxl library, which controls the qemu processes used by VMs. I'd suggest entering a bug at bugzilla.opensuse.org.
I've tried to update libvirt to a newer version using the Open Build Service repos, but then ran into so many conflicting versions that I gave up.
At this point I'm stuck. Does anyone have an idea I can try?
We might be able to eliminate libxl by comparing the interaction with qemu between the working and non-working setups. E.g. enable debug in /etc/libvirt/libvirtd.conf, restart libvirtd, start the offending VM, then look in /var/log/libvirt/libxl/libxl-driver.log for the qemu command line created by libxl and qmp commands sent to qemu. If libxl prepares qemu similarly between the two, then the issue is likely in the qemu package.
Regards, Jim
participants (3)
-
Jim Fehlig -
Ján Tomko -
Marcel Juffermans