[libvirt-users] libvirt-1.2.2-r1 (Gentoo) fails to start LXC containers
I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest available). I have not used LXC containers for a few weeks, so I don't recall what version of libvirt I was using when my container last booted successfully. Unfortunately, Gentoo's portage tree does not offer any previous versions of libvirt that I could downgrade to. TL;DR: My container is configured to use "br0" for its networking. "br0" exists totally inside my linux server - it is NOT bound to any physical NIC. "br0" is used for most of my QEMU and LXC VMs. libvirt is reporting that it cannot find device "veth1". All of my Gentoo packages are up-to-date. Digging through my logs (/var/log/libvirt/libvirt.log), I see that I last successfully booted this LXC container on 2013-07-22, with libvirt reporting version "1.1.0". Thoughts? ostara lxc # equery l libvirt * Searching for libvirt ... [IP-] [ ] app-emulation/libvirt-1.1.2-r1:0 ostara lxc # uname -a Linux ostara 3.10.7-gentoo #2 SMP PREEMPT Sat Aug 24 16:03:57 CDT 2013 x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux ostara ~ # emerge -pvuND world These are the packages that would be merged, in order: Calculating dependencies... done! Total: 0 packages, Size of downloads: 0 kB ostara lxc # virsh -c lxc:/// start dwj-hfax-dev error: Failed to start domain dwj-hfax-dev error: internal error: guest failed to start: PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init error receiving signal from container: Input/output error ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log 2013-09-10 13:56:22.767+0000: starting up PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3 LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log /usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none --handshake 23 --background --veth veth1 PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init error receiving signal from container: Input/output error ostara ~ # tail /var/log/libvirt/libvirtd.log 2013-09-10 13:56:22.998+0000: 12948: info : libvirt version: 1.1.2 2013-09-10 13:56:22.998+0000: 12948: error : virLXCControllerRun:2186 : error receiving signal from container: Input/output error 2013-09-10 13:56:23.053+0000: 12948: error : virCommandWait:2348 : internal error: Child process (ip link del veth1) unexpected exit status 1: Cannot find device "veth1" 2013-09-10 13:56:23.097+0000: 10451: error : virNetSocketReadWire:1369 : Cannot recv data: Connection reset by peer 2013-09-10 13:56:23.153+0000: 10453: error : virLXCProcessStart:1234 : internal error: guest failed to start: PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init error receiving signal from container: Input/output error 2013-09-10 13:56:23.343+0000: 10453: error : virCommandWait:2348 : internal error: Child process (ip link del veth1) unexpected exit status 1: Cannot find device "veth1" (first occurrence of "veth1" in my libvirtd logs) ostara ~ # grep -a veth1 /var/log/libvirt/libvirtd.log | head -1 2013-09-10 13:51:55.333+0000: 9742: error : virCommandWait:2348 : internal error: Child process (ip link del veth1) unexpected exit status 1: Cannot find device "veth1" ostara ~ # which ip /bin/ip ostara ~ # equery b /bin/ip * Searching for /bin/ip ... sys-apps/iproute2-3.8.0 (/bin/ip) ostara ~ # ls -l /bin/ip -rwxr-xr-x 1 root root 282616 May 4 04:08 /bin/ip ostara ~ # ifconfig -a | grep "^[a-z]" br0: flags=4355<UP,BROADCAST,PROMISC,MULTICAST> mtu 1500 br1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500 enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 enp6s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev <domain type='lxc'> <name>dwj-hfax-dev</name> <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/vm/lxc/dwj-hfax-dev'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/usr/portage'/> <target dir='/usr/portage'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/usr/src'/> <target dir='/usr/src'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/home'/> <target dir='/home'/> </filesystem> <interface type='bridge'> <mac address='82:00:00:00:01:01'/> <source bridge='br0'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> </devices> </domain>
On Tue, Sep 10, 2013 at 9:09 AM, Dennis Jenkins <dennis.jenkins.75@gmail.com> wrote:
I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest available). I have not used LXC containers for a few weeks, so I don't recall what version of libvirt I was using when my container last booted successfully.
Unfortunately, Gentoo's portage tree does not offer any previous versions of libvirt that I could downgrade to.
Yeah our security people got a bit over zealous. That's being rectified.
TL;DR: My container is configured to use "br0" for its networking. "br0" exists totally inside my linux server - it is NOT bound to any physical NIC. "br0" is used for most of my QEMU and LXC VMs. libvirt is reporting that it cannot find device "veth1". All of my Gentoo packages are up-to-date.
Digging through my logs (/var/log/libvirt/libvirt.log), I see that I last successfully booted this LXC container on 2013-07-22, with libvirt reporting version "1.1.0".
Thoughts?
You really need to look at /var/log/libvirt/lxc/dwj-hfax-dev.log A suggestion would be to make sure that you have all the necessary kernel options enabled. You can check with: ebuild /usr/portage/app-emulation/libvirt/libvirt-1.1.2-r1.ebuild setup clean -- Doug Goldstein
On Tue, Sep 10, 2013 at 09:09:44AM -0500, Dennis Jenkins wrote:
I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest available). I have not used LXC containers for a few weeks, so I don't recall what version of libvirt I was using when my container last booted successfully.
ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log 2013-09-10 13:56:22.767+0000: starting up PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3 LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log /usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none --handshake 23 --background --veth veth1 PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init error receiving signal from container: Input/output error
We'll need you to increase the log level to get more detail in this file. 1. Stop the 'libvirtd' daemon 2. Edit /etc/libvirt/lxc.conf and set log_with_libvirtd=1 3. Run $ LIBVIRT_LOG_FILTERS=1:lxc LIBVIRT_LOG_OUTPUTS=1:stderr /usr/sbin/libvirtd 4. rm -f /var/log/libvirt/lxc/dwj-hfax-dev.log 5. Try to start your guest with virsh Now send us the much more verbose logfile that was created Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 09/11/2013 04:44 PM, Daniel P. Berrange wrote:
On Tue, Sep 10, 2013 at 09:09:44AM -0500, Dennis Jenkins wrote:
I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest available). I have not used LXC containers for a few weeks, so I don't recall what version of libvirt I was using when my container last booted successfully.
ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log 2013-09-10 13:56:22.767+0000: starting up PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3 LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log /usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none --handshake 23 --background --veth veth1 PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init error receiving signal from container: Input/output error
We'll need you to increase the log level to get more detail in this file.
1. Stop the 'libvirtd' daemon
2. Edit /etc/libvirt/lxc.conf and set log_with_libvirtd=1
3. Run
$ LIBVIRT_LOG_FILTERS=1:lxc LIBVIRT_LOG_OUTPUTS=1:stderr /usr/sbin/libvirtd
4. rm -f /var/log/libvirt/lxc/dwj-hfax-dev.log
5. Try to start your guest with virsh
Now send us the much more verbose logfile that was created
Dennis has already knew what's wrong with his environment, his kernel doesn't support securityfs. libvirt should handle this problem.
On Wed, Sep 11, 2013 at 3:52 AM, Gao feng <gaofeng@cn.fujitsu.com> wrote:
On 09/11/2013 04:44 PM, Daniel P. Berrange wrote:
On Tue, Sep 10, 2013 at 09:09:44AM -0500, Dennis Jenkins wrote:
I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest available). I have not used LXC containers for a few weeks, so I don't recall what version of libvirt I was using when my container last booted successfully.
We'll need you to increase the log level to get more detail in this file.
1. Stop the 'libvirtd' daemon
2. Edit /etc/libvirt/lxc.conf and set log_with_libvirtd=1
3. Run
$ LIBVIRT_LOG_FILTERS=1:lxc LIBVIRT_LOG_OUTPUTS=1:stderr /usr/sbin/libvirtd
4. rm -f /var/log/libvirt/lxc/dwj-hfax-dev.log
5. Try to start your guest with virsh
Now send us the much more verbose logfile that was created
Dennis has already knew what's wrong with his environment, his kernel doesn't support securityfs. libvirt should handle this problem.
I already have logs with that level of debug logging: ostara ~ # grep "log_with_libvirtd" /etc/libvirt/lxc.conf log_with_libvirtd = 1 If it would be useful, I can revert my kernel, reboot, and run libvirtd with the extra env vars set, as listed above. Just let me know. :) I still have the logs from before I upgrades to "libvirt-1.1.2-r1" (lxc worked for me), and after (libvirt failed). I've had the extra logging turned on since last spring when you guys helped me then. There is not much interesting (to me anyway) in the log file: ### last known LXC boot before it broke 2013-07-31 12:49:08.362+0000: starting up PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3 LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log /usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 24 --security=none --handshake 27 --background --veth veth1 PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init ### first attempt to boot LXC after upgrading my kernel and libvirt: 2013-09-10 13:51:54.977+0000: starting up PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3 LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log /usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 21 --security=none --handshake 24 --background --veth veth1 PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init error receiving signal from container: Input/output error
On Wed, Sep 11, 2013 at 3:52 AM, Gao feng <gaofeng@cn.fujitsu.com> wrote:
On 09/11/2013 04:44 PM, Daniel P. Berrange wrote:
On Tue, Sep 10, 2013 at 09:09:44AM -0500, Dennis Jenkins wrote:
I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest available). I have not used LXC containers for a few weeks, so I don't recall what version of libvirt I was using when my container last booted successfully.
ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log 2013-09-10 13:56:22.767+0000: starting up PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3 LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log /usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none --handshake 23 --background --veth veth1 PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3 LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init error receiving signal from container: Input/output error
We'll need you to increase the log level to get more detail in this file.
1. Stop the 'libvirtd' daemon
2. Edit /etc/libvirt/lxc.conf and set log_with_libvirtd=1
3. Run
$ LIBVIRT_LOG_FILTERS=1:lxc LIBVIRT_LOG_OUTPUTS=1:stderr /usr/sbin/libvirtd
4. rm -f /var/log/libvirt/lxc/dwj-hfax-dev.log
5. Try to start your guest with virsh
Now send us the much more verbose logfile that was created
Dennis has already knew what's wrong with his environment, his kernel doesn't support securityfs. libvirt should handle this problem.
Its 6807238d that made securityfs required on the host kernel when userns isn't used. -- Doug Goldstein
participants (4)
-
Daniel P. Berrange -
Dennis Jenkins -
Doug Goldstein -
Gao feng