[libvirt] CreateMachine: Input/output error
Hi! Sometimes libvirt (1.2.7) becomes unable to start any container. Logs show only: error : virDBusCall:1429 : error from service: CreateMachine: Input/output error It looks like dbus_connection_send_with_reply_and_block() returns EIO. Has anyone else seen this kind of issue? I'm currently a bit puzzled where to look for the root cause. Maybe it is related to dbus. -- Thanks, //richard
Hi,
-----Original Message----- From: libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com] On Behalf Of Richard Weinberger Sent: Friday, September 26, 2014 4:07 PM To: libvir-list@redhat.com Subject: [libvirt] CreateMachine: Input/output error
Hi!
Sometimes libvirt (1.2.7) becomes unable to start any container. Logs show only: error : virDBusCall:1429 : error from service: CreateMachine: Input/output error It looks like dbus_connection_send_with_reply_and_block() returns EIO.
Has anyone else seen this kind of issue? I'm currently a bit puzzled where to look for the root cause. Maybe it is related to dbus.
Could you share your XML config? Guess it's something with systemd. Thanks, - Chen
Chen, Am 26.09.2014 10:23, schrieb Chen, Hanxiao:
Has anyone else seen this kind of issue? I'm currently a bit puzzled where to look for the root cause. Maybe it is related to dbus.
Could you share your XML config? Guess it's something with systemd.
There you go: <domain type='lxc'> <name>c_secret_name</name> <memory>524288</memory> <os> <type>exe</type> <init>/sbin/init</init> </os> <idmap> <!-- here be dragons, the mapping is non-linear --> <uid start='0' target='101000' count='998'/> <gid start='0' target='101000' count='998'/> <uid start='65533' target='101998' count='2'/> <gid start='65533' target='101998' count='2'/> </idmap> <devices> <console type='pty'/> <filesystem type='mount'> <source dir='/srv/container0/secret_name/rootfs'/> <target dir='/'/> </filesystem> <interface type='bridge'> <source bridge='br0'/> <mac address='4a:19:0a:XX:XX:XX'/> </interface> </devices> </domain> Nothing special. My host is openSUSE 13.1. Thanks, //richard
Hi Richard,
-----Original Message----- From: Richard Weinberger [mailto:richard@nod.at] Sent: Friday, September 26, 2014 4:59 PM To: Chen, Hanxiao/陈 晗霄; Richard Weinberger; libvir-list@redhat.com Subject: Re: [libvirt] CreateMachine: Input/output error
Chen,
Am 26.09.2014 10:23, schrieb Chen, Hanxiao:
Has anyone else seen this kind of issue? I'm currently a bit puzzled where to look for the root cause. Maybe it is related to dbus.
Could you share your XML config? Guess it's something with systemd.
There you go: <domain type='lxc'> <name>c_secret_name</name> <memory>524288</memory> <os> <type>exe</type> <init>/sbin/init</init> </os> <idmap> <!-- here be dragons, the mapping is non-linear --> <uid start='0' target='101000' count='998'/> <gid start='0' target='101000' count='998'/> <uid start='65533' target='101998' count='2'/> <gid start='65533' target='101998' count='2'/> </idmap> <devices> <console type='pty'/> <filesystem type='mount'> <source dir='/srv/container0/secret_name/rootfs'/> <target dir='/'/> </filesystem> <interface type='bridge'> <source bridge='br0'/> <mac address='4a:19:0a:XX:XX:XX'/> </interface> </devices> </domain>
Nothing special. My host is openSUSE 13.1.
On fedora20 with systemd 208, upstream libvirt, I could reproduce it. It not happened 100%, but once it happened, the container could not be started anymore. One workaround is to undefine it and change another name. With a quick look, I did not find out a explanations. Thanks, - Chen
Chen, Am 26.09.2014 11:49, schrieb Chen, Hanxiao:
Hi Richard,
-----Original Message----- From: Richard Weinberger [mailto:richard@nod.at] Sent: Friday, September 26, 2014 4:59 PM To: Chen, Hanxiao/陈 晗霄; Richard Weinberger; libvir-list@redhat.com Subject: Re: [libvirt] CreateMachine: Input/output error
Chen,
Am 26.09.2014 10:23, schrieb Chen, Hanxiao:
Has anyone else seen this kind of issue? I'm currently a bit puzzled where to look for the root cause. Maybe it is related to dbus.
Could you share your XML config? Guess it's something with systemd.
There you go: <domain type='lxc'> <name>c_secret_name</name> <memory>524288</memory> <os> <type>exe</type> <init>/sbin/init</init> </os> <idmap> <!-- here be dragons, the mapping is non-linear --> <uid start='0' target='101000' count='998'/> <gid start='0' target='101000' count='998'/> <uid start='65533' target='101998' count='2'/> <gid start='65533' target='101998' count='2'/> </idmap> <devices> <console type='pty'/> <filesystem type='mount'> <source dir='/srv/container0/secret_name/rootfs'/> <target dir='/'/> </filesystem> <interface type='bridge'> <source bridge='br0'/> <mac address='4a:19:0a:XX:XX:XX'/> </interface> </devices> </domain>
Nothing special. My host is openSUSE 13.1.
On fedora20 with systemd 208, upstream libvirt, I could reproduce it.
We're also on systemd 208.
It not happened 100%,
Here it happened only twice within months. Always in production, never on my testbed. :(
but once it happened, the container could not be started anymore. One workaround is to undefine it and change another name. With a quick look, I did not find out a explanations.
Hmm, maybe systemd-machined did not cleanup everything upon container exit. Thanks, //richard
Chen, Am 26.09.2014 11:54, schrieb Richard Weinberger:
On fedora20 with systemd 208, upstream libvirt, I could reproduce it.
We're also on systemd 208.
I have an idea, maybe we need this commit in our systemd: http://lists.freedesktop.org/archives/systemd-commits/2014-July/006543.html It is on systemd since v215. Thanks, //richard
Hi,
-----Original Message----- From: Richard Weinberger [mailto:richard@nod.at] Sent: Friday, September 26, 2014 6:31 PM To: Chen, Hanxiao/陈 晗霄; libvir-list@redhat.com Subject: Re: [libvirt] CreateMachine: Input/output error
Chen,
Am 26.09.2014 11:54, schrieb Richard Weinberger:
On fedora20 with systemd 208, upstream libvirt, I could reproduce it.
We're also on systemd 208.
I have an idea, maybe we need this commit in our systemd: http://lists.freedesktop.org/archives/systemd-commits/2014-July/006543.html It is on systemd since v215.
I'm not sure this commit could help because reproduce this issue looks like so unpredictable. I did some tests in the last weekend, unfortunately, I could not reproduce it again with both 208 and 215... Thanks, - Chen
Am 29.09.2014 11:13, schrieb Chen, Hanxiao:
I'm not sure this commit could help because reproduce this issue looks like so unpredictable.
Yeah, maybe.
I did some tests in the last weekend, unfortunately, I could not reproduce it again with both 208 and 215...
Same here. So far I was unable to reproduce it on my testbed. :-\ Thanks, //richard
On Fri, Sep 26, 2014 at 10:06:39AM +0200, Richard Weinberger wrote:
Hi!
Sometimes libvirt (1.2.7) becomes unable to start any container. Logs show only: error : virDBusCall:1429 : error from service: CreateMachine: Input/output error It looks like dbus_connection_send_with_reply_and_block() returns EIO.
Has anyone else seen this kind of issue? I'm currently a bit puzzled where to look for the root cause.
Yeah, some of systemd error messages are kind of unhelpful. Your best bet is probably to strace the 'systemd-machined' process while you create it. If you're lucky one of the strings you see will have the real original error message in it. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Fri, Sep 26, 2014 at 10:06:39AM +0200, Richard Weinberger wrote:
Hi!
Sometimes libvirt (1.2.7) becomes unable to start any container. Logs show only: error : virDBusCall:1429 : error from service: CreateMachine: Input/output error It looks like dbus_connection_send_with_reply_and_block() returns EIO.
Has anyone else seen this kind of issue? I'm currently a bit puzzled where to look for the root cause. Maybe it is related to dbus.
I've seen this while cooking up https://www.redhat.com/archives/libvir-list/2014-September/msg01549.html once. The machine didn't get listed anymore with machinectl, there were no cgroups left but using systemctl I could still see a scope named after that machine like machine-qemu\x2<vmname>.scope failed failed I didn't manage to get rid of that one besides booting. Cheers, -- Guido
Am 26.09.2014 19:40, schrieb Guido Günther:
On Fri, Sep 26, 2014 at 10:06:39AM +0200, Richard Weinberger wrote:
Hi!
Sometimes libvirt (1.2.7) becomes unable to start any container. Logs show only: error : virDBusCall:1429 : error from service: CreateMachine: Input/output error It looks like dbus_connection_send_with_reply_and_block() returns EIO.
Has anyone else seen this kind of issue? I'm currently a bit puzzled where to look for the root cause. Maybe it is related to dbus.
I've seen this while cooking up
https://www.redhat.com/archives/libvir-list/2014-September/msg01549.html
once. The machine didn't get listed anymore with machinectl, there were no cgroups left but using systemctl I could still see a scope named after that machine like
machine-qemu\x2<vmname>.scope failed failed
I didn't manage to get rid of that one besides booting. Cheers,
What systemd version did you use? Thanks, //richard
On Sat, Sep 27, 2014 at 12:04:50AM +0200, Richard Weinberger wrote:
Am 26.09.2014 19:40, schrieb Guido Günther:
On Fri, Sep 26, 2014 at 10:06:39AM +0200, Richard Weinberger wrote:
Hi!
Sometimes libvirt (1.2.7) becomes unable to start any container. Logs show only: error : virDBusCall:1429 : error from service: CreateMachine: Input/output error It looks like dbus_connection_send_with_reply_and_block() returns EIO.
Has anyone else seen this kind of issue? I'm currently a bit puzzled where to look for the root cause. Maybe it is related to dbus.
I've seen this while cooking up
https://www.redhat.com/archives/libvir-list/2014-September/msg01549.html
once. The machine didn't get listed anymore with machinectl, there were no cgroups left but using systemctl I could still see a scope named after that machine like
machine-qemu\x2<vmname>.scope failed failed
I didn't manage to get rid of that one besides booting. Cheers,
What systemd version did you use?
Debian's 208-8 Cheers, -- Guido
participants (5)
-
Chen, Hanxiao -
Daniel P. Berrange -
Guido Günther -
Richard Weinberger -
Richard Weinberger