[libvirt] libvirt failed to spawn with namespace
Hi Michal Build with the latest libvirt source code, I found failed to spawn a qemu process, not sure if something wrong with my environment, can you please help to take a look at it? After build the latest (3.0.0) libvirt, found failed to start an existed domain. root@s2600wt:/home/taget/qemu# virsh start kvm02 error: Failed to start domain kvm02 error: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Unable to move /dev/mqueue mount: Invalid argument then I found you had some code get merged, and required to use /dev/mqueue so mount it then start the domain again: mount -t mqueue none /dev/mqueue root@s2600wt:/home/taget/qemu# virsh start kvm02 error: Failed to start domain kvm02 error: An error occurred, but the cause is unknown some debug log are: … 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakePathHelper:2899 : path=/var/run/libvirt/qemu/kvm02.dev/ mode=0777 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakeParentPath:2971 : path=/var/run/libvirt/qemu/kvm02.dev//hpet 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakePathHelper:2899 : path=/var/run/libvirt/qemu/kvm02.dev/ mode=0777 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakeParentPath:2971 : path=/var/run/libvirt/qemu/kvm02.dev//vfio/vfio 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakePathHelper:2899 : path=/var/run/libvirt/qemu/kvm02.dev//vfio mode=0777 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakePathHelper:2899 : path=/var/run/libvirt/qemu/kvm02.dev/ mode=0777 2017-01-05 07:48:44.195+0000: 37227: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f7dc812fad0 2017-01-05 07:48:44.195+0000: 37227: debug : qemuDomainSetupAllDisks:7076 : Setting up disks 2017-01-05 07:48:44.195+0000: 37227: debug : qemuDomainSetupAllDisks:7085 : Setup all disks 2017-01-05 07:48:44.195+0000: 37227: debug : qemuDomainSetupAllHostdevs:7124 : Setting up hostdevs 2017-01-05 07:48:44.195+0000: 37227: debug : qemuDomainSetupAllHostdevs:7131 : Setup all hostdevs 2017-01-05 07:48:44.195+0000: 37227: debug : qemuDomainSetupAllChardevs:7155 : Setting up chardevs 2017-01-05 07:48:44.195+0000: 37227: debug : qemuDomainSetupAllChardevs:7163 : Setup all chardevs 2017-01-05 07:48:44.195+0000: 37227: debug : qemuDomainSetupAllInputs:7231 : Setting up disks 2017-01-05 07:48:44.195+0000: 37227: debug : qemuDomainSetupAllInputs:7238 : Setup all disks 2017-01-05 07:48:44.195+0000: 37227: debug : qemuDomainSetupAllRNGs:7270 : Setting up RNGs 2017-01-05 07:48:44.195+0000: 37227: debug : qemuDomainSetupAllRNGs:7278 : Setup all RNGs 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakePathHelper:2899 : path=/dev/pts mode=0777 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakePathHelper:2899 : path=/dev mode=0777 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakePathHelper:2899 : path=/dev/shm mode=0777 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakePathHelper:2899 : path=/dev mode=0777 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakePathHelper:2899 : path=/dev/mqueue mode=0777 2017-01-05 07:48:44.195+0000: 37227: debug : virFileMakePathHelper:2899 : path=/dev mode=0777 2017-01-05 07:48:44.195+0000: 37227: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f7dc812fad0 2017-01-05 07:48:44.195+0000: 37227: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f7dc812fad0 2017-01-05 07:48:44.195+0000: 37227: debug : qemuProcessHook:2693 : Hook complete ret=0 2017-01-05 07:48:44.195+0000: 37227: debug : virExec:699 : Done hook 0 2017-01-05 07:48:44.195+0000: 37227: debug : virExec:736 : Setting child uid:gid to 0:0 with caps 0 2017-01-05 07:48:44.195+0000: 37227: debug : virCommandHandshakeChild:435 : Notifying parent for handshake start on 25 2017-01-05 07:48:44.195+0000: 37227: debug : virCommandHandshakeChild:443 : Waiting on parent for handshake complete on 26 libvirt: error : libvirtd quit during handshake: Input/output error 2017-01-05 07:48:44.216+0000: shutting down, reason=failed /\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ I work on the latest libvirt git tree. commit 866641d4c5706413393913fdb3bb1cd077683d21 Author: Michal Privoznik <mprivozn@redhat.com> Date: Sat Dec 24 17:55:48 2016 +0100 NEWS: Update after qemu namespace fix Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Best Regards Eli Qiao(乔立勇）OpenStack Core team OTC Intel. --
On 01/05/2017 08:48 AM, Qiao, Liyong wrote:
Hi Michal
Build with the latest libvirt source code, I found failed to spawn a qemu process, not sure if something wrong with my environment, can you please help to take a look at it?
After build the latest (3.0.0) libvirt, found failed to start an existed domain.
root@s2600wt:/home/taget/qemu# virsh start kvm02 error: Failed to start domain kvm02 error: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Unable to move /dev/mqueue mount: Invalid argument
D'oh. There's a list of FSs that libvirt tries to preserve and bind mount from the parent namespace. But it is not fault tolerant.
then I found you had some code get merged, and required to use /dev/mqueue
Yes. Exactly.
so mount it then start the domain again:
mount -t mqueue none /dev/mqueue
root@s2600wt:/home/taget/qemu# virsh start kvm02 error: Failed to start domain kvm02 error: An error occurred, but the cause is unknown
Can you get /var/log/libvirt/qemu/kvm02.log? As a workaround you can set namespaces=[] in qemu.conf, but that is really meant just like a workaround until all of these edge cases are fixed. Meanwhile, I started working on a fix that will drop the persistent list and parse /proc/mounts instead. Michal
Hi Michal Thanks for your reply, the workaround works well for me. Recreate kvm02.log and attach it for you. BTW, can you help me to understand how can you make this [1] work if it’s /dev/mqueue. As far as I know /dev/mqueue is not a block device, how can we mount /dev/mqueue to somewhere ? [1] https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_domain.c;h=67e... Best Regards Eli Qiao(乔立勇）OpenStack Core team OTC Intel. -- On 05/01/2017, 4:34 PM, "Michal Privoznik" <mprivozn@redhat.com> wrote: On 01/05/2017 08:48 AM, Qiao, Liyong wrote: > Hi Michal > > Build with the latest libvirt source code, I found failed to spawn a qemu process, not sure if something wrong with my environment, can you please help to take a look at it? > > After build the latest (3.0.0) libvirt, found failed to start an existed domain. > > root@s2600wt:/home/taget/qemu# virsh start kvm02 > error: Failed to start domain kvm02 > error: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Unable to move /dev/mqueue mount: Invalid argument D'oh. There's a list of FSs that libvirt tries to preserve and bind mount from the parent namespace. But it is not fault tolerant. > > then I found you had some code get merged, and required to use /dev/mqueue Yes. Exactly. > > so mount it then start the domain again: > > mount -t mqueue none /dev/mqueue > > root@s2600wt:/home/taget/qemu# virsh start kvm02 > error: Failed to start domain kvm02 > error: An error occurred, but the cause is unknown Can you get /var/log/libvirt/qemu/kvm02.log? As a workaround you can set namespaces=[] in qemu.conf, but that is really meant just like a workaround until all of these edge cases are fixed. Meanwhile, I started working on a fix that will drop the persistent list and parse /proc/mounts instead. Michal
On 01/05/2017 09:51 AM, Qiao, Liyong wrote:
Hi Michal Thanks for your reply, the workaround works well for me. Recreate kvm02.log and attach it for you.
Ah sorry. For some reason I thought that the first log you've sent is daemon log. It wasn't and in fact I'd like to see it for the case where you mount /dev/mqueue and enable the namespace (=undo the workaround).
BTW, can you help me to understand how can you make this [1] work if it’s /dev/mqueue. As far as I know /dev/mqueue is not a block device, how can we mount /dev/mqueue to somewhere ?
A filesystem doesn't have to be backed by a block device in order to be mountable. tmpfs is a great example. mqueue is yet another one of such filesystems. BTW: $ grep nodev /proc/filesystems Looks like there's a lot of nodev filesystems. Michal
On 01/05/2017 11:55 AM, Michal Privoznik wrote:
On 01/05/2017 09:51 AM, Qiao, Liyong wrote:
Hi Michal Thanks for your reply, the workaround works well for me. Recreate kvm02.log and attach it for you.
Ah sorry. For some reason I thought that the first log you've sent is daemon log. It wasn't and in fact I'd like to see it for the case where you mount /dev/mqueue and enable the namespace (=undo the workaround).
BTW, can you help me to understand how can you make this [1] work if it’s /dev/mqueue. As far as I know /dev/mqueue is not a block device, how can we mount /dev/mqueue to somewhere ?
A filesystem doesn't have to be backed by a block device in order to be mountable. tmpfs is a great example. mqueue is yet another one of such filesystems. BTW:
$ grep nodev /proc/filesystems
Looks like there's a lot of nodev filesystems.
BTW: I've just posted a patch for this issue. Can you give it a try please? https://www.redhat.com/archives/libvir-list/2017-January/msg00139.html Michal
participants (2)
-
Michal Privoznik -
Qiao, Liyong