[libvirt-users] Unable to start LXC on Gentoo w/ libvirt 1.0.4 or 1.0.5. 1.0.3 works
Hello. I use libvirtd on my Gentoo development system to manage both QEMU and LXC. When 1.0.3 came out, I updated to it from 1.0.3-r2, but 1.0.4 failed to start my LXC containers. I did not research the issue at the time, so I revert to 1.0.3-r2. Today I updated to 1.0.5 and my LXC containers still fail to start. I have not changed my domain XML at all. I am looking for suggestions on what to try to resolve the problem. Maybe I'm missing a package or have something mus-configured for libvirt-1.0.5. btw, A month ago I updates to the most recent "udev" (udev-200). This required me to rename all of my network interfaces. My QEMU and LXC containers continued to work. All of my VMs use bridged networking on "br0" or "br1". Those virtual interfaces are not bridged to any physical NIC. They exist only inside my host's kernel. I have extensive iptables rules to filter traffic between my network segments (physical + virtual). TL;DR: this was burried in a log file: " error : lxcContainerMountFSDevPTS:808 : Failed to mount /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts on /dev/pts: No such file or directory". But that directory DOES exist in the container: ostara ~ # ls -ld /vm/lxc/dwj-lnx-dev/dev/pts/ drwxr-xr-x 2 root root 4096 Apr 27 2011 /vm/lxc/dwj-lnx-dev/dev/pts/ If I revert to libvirt-1.0.3-r2, I can boot my LXC containers again. But if possible, I'd like to keep my system current. I've ran the usual Gentoo tools to verify that all package dependencies are installed, revdep-rebuild reports that all ELFs link against the proper shared objects. ostara ~ # virsh --version 1.0.5 ostara ~ # equery l libvirt * Searching for libvirt ... [IP-] [ ] app-emulation/libvirt-1.0.5:0 ostara ~ # equery -q u libvirt | xargs echo -audit -avahi +caps -firewalld -fuse -iscsi +libvirtd +lvm +lxc +macvtap +nfs +nls -numa -openvz -parted +pcap -phyp -policykit -python +qemu -rbd +sasl +udev +uml +vepa +virt-network -virtualbox -xen ostara ~ # virsh -c lxc:/// list --all Id Name State ---------------------------------------------------- - dwj-lnx-dev shut off - vm1 shut off ostara ~ # virsh -c lxc:/// start dwj-lnx-dev error: Failed to start domain dwj-lnx-dev error: internal error guest failed to start: PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=fbcd8c3a-9939-12b4-727d-5d3526bc448f LIBVIRT_LXC_UUID=fbcd8c3a-9939-12b4-727d-5d3526bc448f LIBVIRT_LXC_NAME=dwj-lnx-dev /sbin/init error receiving signal from container: Input/output error ostara ~ # tail -n 10 /var/log/libvirt/libvirtd.log 2013-05-14 13:03:47.549+0000: 22352: info : libvirt version: 1.0.5 2013-05-14 13:03:47.549+0000: 22352: error : virDBusGetSystemBus:86 : internal error Unable to get DBus system bus connection: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory 2013-05-14 13:03:47.549+0000: 22352: warning : ebiptablesDriverInitCLITools:4233 : Could not find 'ip6tables' executable 2013-05-14 13:04:21.629+0000: 22341: error : virNetSocketReadWire:1362 : Cannot recv data: Connection reset by peer 2013-05-14 13:04:21.729+0000: 22345: error : virLXCProcessStart:1285 : internal error guest failed to start: PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=fbcd8c3a-9939-12b4-727d-5d3526bc448f LIBVIRT_LXC_UUID=fbcd8c3a-9939-12b4-727d-5d3526bc448f LIBVIRT_LXC_NAME=dwj-lnx-dev /sbin/init error receiving signal from container: Input/output error 2013-05-14 13:04:21.731+0000: 22345: error : virNetDevSetOnline:544 : Cannot get interface flags on 'veth0': No such device 2013-05-14 13:04:21.778+0000: 22345: error : virCommandWait:2353 : internal error Child process (ip link del veth0) unexpected exit status 1: Cannot find device "veth0" 2013-05-14 13:04:21.917+0000: 22345: error : virCommandWait:2353 : internal error Child process (ip link del veth1) unexpected exit status 1: Cannot find device "veth1" ostara ~ # tail -n 10 /var/log/libvirt/lxc/dwj-lnx-dev.log 2013-05-14 13:04:20.894+0000: starting up PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.2 LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/libvirt_lxc --name dwj-lnx-dev --console 22 --security=none --handshake 25 --background --veth veth1 PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=fbcd8c3a-9939-12b4-727d-5d3526bc448f LIBVIRT_LXC_UUID=fbcd8c3a-9939-12b4-727d-5d3526bc448f LIBVIRT_LXC_NAME=dwj-lnx-dev /sbin/init 2013-05-14 13:04:21.562+0000: 1: info : libvirt version: 1.0.5 2013-05-14 13:04:21.562+0000: 1: error : lxcContainerMountFSDevPTS:808 : Failed to mount /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts on /dev/pts: No such file or directory 2013-05-14 13:04:21.562+0000: 22562: info : libvirt version: 1.0.5 2013-05-14 13:04:21.562+0000: 22562: error : virLXCControllerRun:1481 : error receiving signal from container: Input/output error error receiving signal from container: Input/output error 2013-05-14 13:04:21.628+0000: 22562: error : virCommandWait:2353 : internal error Child process (ip link del veth1) unexpected exit status 1: Cannot find device "veth1" ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev <domain type='lxc'> <name>dwj-lnx-dev</name> <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid> <memory unit='KiB'>500000</memory> <currentMemory unit='KiB'>500000</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/vm/lxc/dwj-lnx-dev'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/usr/portage'/> <target dir='/usr/portage'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/usr/src'/> <target dir='/usr/src'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/home'/> <target dir='/home'/> </filesystem> <interface type='bridge'> <mac address='82:00:00:00:01:00'/> <source bridge='br0'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> </devices> </domain> ostara ~ # ifconfig -a | egrep -v "(RX|TX) (packets|errors)" br0: flags=4355<UP,BROADCAST,PROMISC,MULTICAST> mtu 1500 inet 192.168.2.1 netmask 255.255.255.0 broadcast 192.168.2.255 ether 00:00:00:00:00:00 txqueuelen 0 (Ethernet) br1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500 inet 192.168.3.1 netmask 255.255.255.0 broadcast 192.168.3.255 ether 06:9b:23:fb:14:5f txqueuelen 0 (Ethernet) enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.1 netmask 255.255.255.0 broadcast 192.168.0.255 ether bc:ae:c5:16:6f:3b txqueuelen 1000 (Ethernet) device interrupt 18 memory 0xfade0000-fae00000 enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet xx.xx.xx.xx netmask 255.255.252.0 broadcast xx.xx.xx.xx ether bc:ae:c5:16:70:3f txqueuelen 1000 (Ethernet) device interrupt 19 memory 0xfaee0000-faf00000 enp6s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.4.1 netmask 255.255.255.0 broadcast 192.168.4.255 ether 00:03:6d:14:78:30 txqueuelen 1000 (Ethernet) lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 loop txqueuelen 0 (Local Loopback) virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255 ether c2:ff:56:b7:a7:cd txqueuelen 0 (Ethernet)
On Tue, May 14, 2013 at 08:28:48AM -0500, Dennis Jenkins wrote:
I use libvirtd on my Gentoo development system to manage both QEMU and LXC. When 1.0.3 came out, I updated to it from 1.0.3-r2, but 1.0.4 failed to start my LXC containers. I did not research the issue at the time, so I revert to 1.0.3-r2. Today I updated to 1.0.5 and my LXC containers still fail to start. I have not changed my domain XML at all.
TL;DR: this was burried in a log file: " error : lxcContainerMountFSDevPTS:808 : Failed to mount /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts on /dev/pts: No such file or directory". But that directory DOES exist in the container:
ostara ~ # ls -ld /vm/lxc/dwj-lnx-dev/dev/pts/ drwxr-xr-x 2 root root 4096 Apr 27 2011 /vm/lxc/dwj-lnx-dev/dev/pts/
Actually libvirt will mount a brand new tmpfs at /dev inside the container, so whatever your filesystem has at /dev is not used. Libvirt should of course also be creating the /dev/pts mount point. This makes me think the error message could actually refer to the "/.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts" path.
ostara ~ # tail -n 10 /var/log/libvirt/lxc/dwj-lnx-dev.log 2013-05-14 13:04:20.894+0000: starting up PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.2 LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/libvirt_lxc --name dwj-lnx-dev --console 22 --security=none --handshake 25 --background --veth veth1 PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=fbcd8c3a-9939-12b4-727d-5d3526bc448f LIBVIRT_LXC_UUID=fbcd8c3a-9939-12b4-727d-5d3526bc448f LIBVIRT_LXC_NAME=dwj-lnx-dev /sbin/init 2013-05-14 13:04:21.562+0000: 1: info : libvirt version: 1.0.5 2013-05-14 13:04:21.562+0000: 1: error : lxcContainerMountFSDevPTS:808 : Failed to mount /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts on /dev/pts: No such file or directory 2013-05-14 13:04:21.562+0000: 22562: info : libvirt version: 1.0.5 2013-05-14 13:04:21.562+0000: 22562: error : virLXCControllerRun:1481 : error receiving signal from container: Input/output error error receiving signal from container: Input/output error 2013-05-14 13:04:21.628+0000: 22562: error : virCommandWait:2353 : internal error Child process (ip link del veth1) unexpected exit status 1: Cannot find device "veth1"
We need to get more debug info from this file. Stop libvirtd and edit /etc/libvirt/lxc.conf and set log_with_libvirtd=1 and then run libvirtd with LIBVIRT_LOG_OUTPUTS=1:stderr LIBVIRT_LOG_FILTERS="3:event 3:json 3:object 1:util 1:lxc" /usr/sbin/libvirtd and then try to start a container again. The container log file above should now get much more verbose data (i don't care about the log output from libvirtd itself - just the container log file). Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Hello Daniel, Thank you for your quick response. The logs you requested are below. On Tue, May 14, 2013 at 8:55 AM, Daniel P. Berrange <berrange@redhat.com>wrote:
On Tue, May 14, 2013 at 08:28:48AM -0500, Dennis Jenkins wrote:We need to get more debug info from this file. Stop libvirtd and edit
/etc/libvirt/lxc.conf
and set
log_with_libvirtd=1
and then run libvirtd with
LIBVIRT_LOG_OUTPUTS=1:stderr LIBVIRT_LOG_FILTERS="3:event 3:json 3:object 1:util 1:lxc" /usr/sbin/libvirtd
Done.
and then try to start a container again. The container log file above should now get much more verbose data (i don't care about the log output from libvirtd itself - just the container log file).
Daniel
ostara ~ # virsh -c lxc:/// start dwj-lnx-dev error: Failed to start domain dwj-lnx-dev error: internal error guest failed to start: ' ' 2013-05-14 14:01:24.666+0000: 20695: The log is large to place inline in an email (32K), so I posted it (no expiration) to pastebin. But below are the juicy bits. If you want the entire log in the email, I will resend it. http://pastebin.com/nXcNQukz 2013-05-14 14:01:25.265+0000: 1: debug : lxcContainerMountBasicFS:712 : Mount sysfs on /sys type=sysfs flags=1021, opts=(null) 2013-05-14 14:01:25.265+0000: 1: debug : lxcContainerMountBasicFS:736 : Mount devfs on /dev type=tmpfs flags=2, opts=mode=755,size=65536 2013-05-14 14:01:25.265+0000: 1: debug : lxcContainerMountBasicFS:748 : rc=0 2013-05-14 14:01:25.265+0000: 1: debug : virFileMakePathHelper:1274 : path=/dev/pts mode=0777 2013-05-14 14:01:25.265+0000: 1: debug : virFileMakePathHelper:1274 : path=/dev mode=0777 2013-05-14 14:01:25.265+0000: 1: debug : lxcContainerMountFSDevPTS:802 : Trying to move /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts to /dev/pts 2013-05-14 14:01:25.265+0000: 1: error : lxcContainerMountFSDevPTS:808 : Failed to mount /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts on /dev/pts: No such file or directory 2013-05-14 14:01:25.265+0000: 1: debug : virFileClose:72 : Closed fd 13 2013-05-14 14:01:25.265+0000: 1: debug : virFileClose:72 : Closed fd 9 2013-05-14 14:01:25.265+0000: 1: debug : virFileClose:72 : Closed fd 11 2013-05-14 14:01:25.265+0000: 20717: error : virLXCControllerRun:1481 : error receiving signal from container: Input/output error 2013-05-14 14:01:25.265+0000: 20717: debug : virFileClose:72 : Closed fd 8 2013-05-14 14:01:25.265+0000: 20717: debug : virFileClose:72 : Closed fd 10 2013-05-14 14:01:25.265+0000: 20717: debug : virProcessAbort:96 : aborting child process 20718
On Tue, May 14, 2013 at 09:06:36AM -0500, Dennis Jenkins wrote:
The log is large to place inline in an email (32K), so I posted it (no expiration) to pastebin. But below are the juicy bits. If you want the entire log in the email, I will resend it.
2013-05-14 14:01:25.265+0000: 1: debug : lxcContainerMountBasicFS:712 : Mount sysfs on /sys type=sysfs flags=1021, opts=(null) 2013-05-14 14:01:25.265+0000: 1: debug : lxcContainerMountBasicFS:736 : Mount devfs on /dev type=tmpfs flags=2, opts=mode=755,size=65536 2013-05-14 14:01:25.265+0000: 1: debug : lxcContainerMountBasicFS:748 : rc=0
2013-05-14 14:01:25.265+0000: 1: debug : virFileMakePathHelper:1274 : path=/dev/pts mode=0777 2013-05-14 14:01:25.265+0000: 1: debug : virFileMakePathHelper:1274 : path=/dev mode=0777 2013-05-14 14:01:25.265+0000: 1: debug : lxcContainerMountFSDevPTS:802 : Trying to move /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts to /dev/pts 2013-05-14 14:01:25.265+0000: 1: error : lxcContainerMountFSDevPTS:808 : Failed to mount /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts on /dev/pts: No such file or directory
Hmm, so from the logs I'm fairly certain the missing path is the /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts one, not /dev/pts Can you tell me if any part of '/var/run/libvirt/lxc/' is a symlink to elsewhere ? In particular is '/var/run' a symlink, and if so is it relative or absolute ? Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, May 14, 2013 at 9:12 AM, Daniel P. Berrange <berrange@redhat.com>wrote:
On Tue, May 14, 2013 at 09:06:36AM -0500, Dennis Jenkins wrote:
The log is large to place inline in an email (32K), so I posted it (no expiration) to pastebin. But below are the juicy bits. If you want the entire log in the email, I will resend it.
Hmm, so from the logs I'm fairly certain the missing path is the /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts one, not /dev/pts
Can you tell me if any part of '/var/run/libvirt/lxc/' is a symlink to elsewhere ? In particular is '/var/run' a symlink, and if so is it relative or absolute ?
No symlinks under "/var/run/libvirt", but "/var/run" is a symlink to "/run". ostara ~ # file /var/run /var/run: symbolic link to `/run' ostara ~ # mount | grep run tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755) ostara ~ # find /var/run/libvirt -ls 599182 0 drwxr-xr-x 8 root root 200 May 14 09:00 /var/run/libvirt 24808928 0 srwxrwxrwx 1 root root 0 May 14 09:00 /var/run/libvirt/libvirt-sock-ro 24808926 0 srwx------ 1 root root 0 May 14 09:00 /var/run/libvirt/libvirt-sock 686482 0 drwxr-xr-x 2 root root 40 Apr 19 12:01 /var/run/libvirt/uml-guest 599187 0 drwxr-xr-x 3 root root 80 May 14 09:01 /var/run/libvirt/lxc 24810554 0 srwx------ 1 root root 0 May 14 09:01 /var/run/libvirt/lxc/dwj-lnx-dev.sock 24712812 0 drwxr-xr-x 2 root root 40 May 14 08:04 /var/run/libvirt/lxc/dwj-lnx-dev.devpts 599186 0 drwxr-xr-x 2 root root 40 Apr 19 11:38 /var/run/libvirt/uml 599185 0 drwxr-xr-x 2 root root 80 May 14 09:00 /var/run/libvirt/network 24808953 0 -rw-r--r-- 1 root root 0 May 14 09:00 /var/run/libvirt/network/nwfilter.leases 24808936 4 -rw-r--r-- 1 root root 6 May 14 09:00 /var/run/libvirt/network/default.pid 599184 0 drwxr-xr-x 2 root root 40 Apr 19 11:38 /var/run/libvirt/lockd 599183 0 drwxr-xr-x 2 root root 40 May 14 08:59 /var/run/libvirt/qemu
On Tue, May 14, 2013 at 09:19:51AM -0500, Dennis Jenkins wrote:
On Tue, May 14, 2013 at 9:12 AM, Daniel P. Berrange <berrange@redhat.com>wrote:
On Tue, May 14, 2013 at 09:06:36AM -0500, Dennis Jenkins wrote:
The log is large to place inline in an email (32K), so I posted it (no expiration) to pastebin. But below are the juicy bits. If you want the entire log in the email, I will resend it.
Hmm, so from the logs I'm fairly certain the missing path is the /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts one, not /dev/pts
Can you tell me if any part of '/var/run/libvirt/lxc/' is a symlink to elsewhere ? In particular is '/var/run' a symlink, and if so is it relative or absolute ?
No symlinks under "/var/run/libvirt", but "/var/run" is a symlink to "/run".
ostara ~ # file /var/run /var/run: symbolic link to `/run'
Ok, that'll be the problem. When libvirt prepends /.oldroot onto the path, the symlink keeps pointing to /run instead of to /.oldroot/run :-( On Fedora "/var/run" is a relative symlink to "../run" to avoid these kinds of problem. As a quick workaround you can make your symlink relative too. I'll get a fix into the next libvirt release to deal with absolute symlinks better. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, May 14, 2013 at 9:27 AM, Daniel P. Berrange <berrange@redhat.com>wrote:
On Tue, May 14, 2013 at 09:19:51AM -0500, Dennis Jenkins wrote:
On Tue, May 14, 2013 at 9:12 AM, Daniel P. Berrange <berrange@redhat.com wrote:
On Tue, May 14, 2013 at 09:06:36AM -0500, Dennis Jenkins wrote:
Hmm, so from the logs I'm fairly certain the missing path is the /.oldroot//var/run/libvirt/lxc/dwj-lnx-dev.devpts one, not /dev/pts
Can you tell me if any part of '/var/run/libvirt/lxc/' is a symlink to elsewhere ? In particular is '/var/run' a symlink, and if so is it relative or absolute ?
No symlinks under "/var/run/libvirt", but "/var/run" is a symlink to "/run".
ostara ~ # file /var/run /var/run: symbolic link to `/run'
Ok, that'll be the problem. When libvirt prepends /.oldroot onto the path, the symlink keeps pointing to /run instead of to /.oldroot/run :-(
On Fedora "/var/run" is a relative symlink to "../run" to avoid these kinds of problem.
As a quick workaround you can make your symlink relative too.
I'll get a fix into the next libvirt release to deal with absolute symlinks better.
ostara ~ # cd /var ostara var # rm run; ln -s ../run run ostara var # virsh -c lxc:/// start dwj-lnx-dev Domain dwj-lnx-dev started Yeah. Diagnosis confirmed. Daniel, thank you for your assistance.
participants (2)
-
Daniel P. Berrange -
Dennis Jenkins