12:29 p.m.
I'm chasing down a very frustrating bug which only happens on i686 &
Koji during the nbdkit tests and seemingly nowhere else. Anyway this
is what I've been able to put together:
The libguestfs appliance (guest) is created with this XML snippet:
<disk device="disk" type="network">
<source protocol="nbd">
<host transport="unix" socket="cow.sock"/>
</source>
<target dev="sda" bus="scsi"/>
<driver name="qemu" type="raw"
cache="writeback"/>
<address type="drive" controller="0" bus="0"
target="0" unit="0"/>
</disk>
It fails to start with the error:
internal error: process exited while connecting to monitor: 2018-06-06T17:02:54.450507Z
qemu-system-i386: -drive
file=nbd+unix://?socket=cow.sock,format=raw,if=none,id=drive-scsi0-0-0-0,cache=writeback:
Failed to connect socket cow.sock: No such file or directory [code=1 int1=-1]
The socket definitely exists in the directory of the program running
libvirt. I verified that by adding ls -l commands to the build:
srwxr-xr-x. 1 mockbuild mockbuild 0 Jun 6 17:02 cow.sock
Permissions on the containing directories are fine (the whole lot is
running as mockbuild):
+ pwd
/builddir/build/BUILD/nbdkit-1.3.1/tests
+ ls -ld /builddir
drwx------. 5 mockbuild mockbuild 4096 Jun 6 17:00 /builddir
+ ls -ld /builddir/build
drwxr-xr-x. 9 mockbuild mockbuild 4096 Jun 6 16:57 /builddir/build
+ ls -ld /builddir/build/BUILD
drwxrwxr-x. 3 mockbuild 1000 4096 Jun 6 16:59 /builddir/build/BUILD
+ ls -ld /builddir/build/BUILD/nbdkit-1.3.1
drwxr-xr-x. 11 mockbuild mockbuild 4096 Jun 6 17:00 /builddir/build/BUILD/nbdkit-1.3.1
+ ls -ld /builddir/build/BUILD/nbdkit-1.3.1/tests
drwxr-xr-x. 4 mockbuild mockbuild 4096 Jun 6 17:02
/builddir/build/BUILD/nbdkit-1.3.1/tests
The qemu log doesn't give any clues:
2018-06-06 17:02:54.412+0000: starting up libvirt version: 4.4.0, package: 1.fc29
(Fedora Project, 2018-06-05-11:53:38, buildvm-12.phx2.fedoraproject.org), qemu version:
2.12.0qemu-2.12.0-2.fc29, kernel: 4.16.8-300.fc28.x86_64, hostname:
buildhw-10.phx2.fedoraproject.org
LC_ALL=C
PATH=/builddir/build/BUILD/nbdkit-1.3.1:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
HOME=/builddir USER=mockbuild LOGNAME=mockbuild QEMU_AUDIO_DRV=none TMPDIR=/var/tmp
/usr/bin/qemu-system-i386 -name guest=guestfs-62ak9dzrjxfoctua,debug-threads=on -S -object
secret,id=masterKey0,format=raw,file=/builddir/.config/libvirt/qemu/lib/domain-3-guestfs-62ak9dzrjxfo/master-key.aes
-machine pc-i440fx-2.12,accel=tcg,usb=off,dump-guest-core=off -m 500 -realtime mlock=off
-smp 1,sockets=1,cores=1,threads=1 -uuid 6094da49-2def-4256-b901-e4fe5f1409c8 -display
none -no-user-config -nodefaults -device sga -chardev
socket,id=charmonitor,path=/builddir/.config/libvirt/qemu/lib/domain-3-guestfs-62ak9dzrjxfo/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global
kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -no-acpi -boot strict=on -kernel
/var/tmp/.guestfs-1000/appliance.d/kernel -initrd
/var/tmp/.guestfs-1000/appliance.d/initrd -append 'panic=1 noapic console=ttyS0
edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1
cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1
root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=vt100' -device
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive
'file=nbd+unix://?socket=cow.sock,format=raw,if=none,id=drive-scsi0-0-0-0,cache=writeback'
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1,write-cache=on
-drive
file=/tmp/libguestfsprC4lc/overlay1.qcow2,format=qcow2,if=none,id=drive-scsi0-0-1-0,cache=unsafe
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0,write-cache=on
-chardev socket,id=charserial0,path=/tmp/libguestfsj35K6W/console.sock -device
isa-serial,chardev=charserial0,id=serial0 -chardev
socket,id=charchannel0,path=/tmp/libguestfsj35K6W/guestfsd.sock -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0
-object rng-random,id=objrng0,filename=/dev/urandom -device
virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x4 -sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
2018-06-06 17:02:54.412+0000: Domain id=3 is tainted: custom-argv
2018-06-06T17:02:54.450507Z qemu-system-i386: -drive
file=nbd+unix://?socket=cow.sock,format=raw,if=none,id=drive-scsi0-0-0-0,cache=writeback:
Failed to connect socket cow.sock: No such file or directory
2018-06-06 17:02:54.453+0000: shutting down, reason=failed
At this point I don't really have any ideas. Does libvirt now run
qemu in a different directory? Does the error message mean something
else apart from the file not existing?
Also the nbd+unix syntax doesn't appear to be documented in qemu.
Is this a new thing? Normally we use nbd:unix:...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
1:25 p.m.
On 06/06/2018 12:29 PM, Richard W.M. Jones wrote:
I'm chasing down a very frustrating bug which only happens on
i686 &
Koji during the nbdkit tests and seemingly nowhere else. Anyway this
is what I've been able to put together:
The libguestfs appliance (guest) is created with this XML snippet:
<disk device="disk" type="network">
<source protocol="nbd">
<host transport="unix" socket="cow.sock"/>
Have you tried an absolute path?
At this point I don't really have any ideas. Does libvirt now
run
qemu in a different directory? Does the error message mean something
else apart from the file not existing?
Libvirtd does chdir("/") early on, so likely qemu is run with / (rather
than the original current working directory) as its root. So an absolute
path should fix things.
2018-06-06T17:02:54.450507Z qemu-system-i386: -drive
file=nbd+unix://?socket=cow.sock,format=raw,if=none,id=drive-scsi0-0-0-0,cache=writeback:
Failed to connect socket cow.sock: No such file or directory
Also the nbd+unix syntax doesn't appear to be documented in qemu.
Is this a new thing? Normally we use nbd:unix:...
nbd:unix:... is the old-style legacy form, nbd+unix:// is the URI style.
The URI style is a bit more flexible (you can add options without
having to add more ad-hoc parsing). Ideally, we'll be moving libvirt to
an even newer -blockdev style (which can either directly use the JSON
you'd hand to QMP blockdev-add, or which uses a dotted syntax similar to
-drive); that's been an ongoing task for Peter. As for lack of
documentation in qemu, I'm not surprised.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
1:35 p.m.
On Wed, Jun 06, 2018 at 01:25:27PM -0500, Eric Blake wrote:
On 06/06/2018 12:29 PM, Richard W.M. Jones wrote:
>I'm chasing down a very frustrating bug which only happens on i686 &
>Koji during the nbdkit tests and seemingly nowhere else. Anyway this
>is what I've been able to put together:
>
>The libguestfs appliance (guest) is created with this XML snippet:
>
> <disk device="disk" type="network">
> <source protocol="nbd">
> <host transport="unix" socket="cow.sock"/>
Have you tried an absolute path?
Yes it works with an absolute path. Isn't this a bug in libvirt?
Also it works on x86_64 and in earlier versions of libvirt.
I'll post my nbdkit workaround soon.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
6:20 a.m.
I don't know whether or not we decided this was a bug, but
I have filed one anyway:
https://bugzilla.redhat.com/show_bug.cgi?id=1588447
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
6:38 a.m.
On Thu, Jun 07, 2018 at 12:20:16 +0100, Richard W.M. Jones wrote:
I don't know whether or not we decided this was a bug, but
I have filed one anyway:
https://bugzilla.redhat.com/show_bug.cgi?id=1588447
Your XML does not conform to the libvirt domain XML schema which always
called for an absolute path as we do with all paths in the domain XML:
<group>
<attribute name="transport">
<value>unix</value>
</attribute>
<attribute name="socket">
<ref name="absFilePath"/>
</attribute>
</group>
We just did not validate this particular code path in the parser itself
(when the domain schema validation is not used).
6:42 a.m.
On Thu, Jun 07, 2018 at 01:38:47PM +0200, Peter Krempa wrote:
On Thu, Jun 07, 2018 at 12:20:16 +0100, Richard W.M. Jones wrote:
>
> I don't know whether or not we decided this was a bug, but
> I have filed one anyway:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1588447
Your XML does not conform to the libvirt domain XML schema which always
called for an absolute path as we do with all paths in the domain XML:
<group>
<attribute name="transport">
<value>unix</value>
</attribute>
<attribute name="socket">
<ref name="absFilePath"/>
</attribute>
</group>
We just did not validate this particular code path in the parser itself
(when the domain schema validation is not used).
Sure. Libvirt does need to validate the paths because it's possible
that the *wrong* path will be opened if a relative path is used,
eg. if a socket with the same name happens to exist in "/" (or
whatever is the working directory that libvirt changes to).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
2360
days inactive
2361
days old
5 comments
3 participants
participants (3)
-
Eric Blake -
Peter Krempa -
Richard W.M. Jones