9:09 a.m.
I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest available). I
have not used LXC containers for a few weeks, so I don't recall what
version of libvirt I was using when my container last booted successfully.
Unfortunately, Gentoo's portage tree does not offer any previous versions
of libvirt that I could downgrade to.
TL;DR: My container is configured to use "br0" for its networking.
"br0"
exists totally inside my linux server - it is NOT bound to any physical
NIC. "br0" is used for most of my QEMU and LXC VMs. libvirt is reporting
that it cannot find device "veth1". All of my Gentoo packages are
up-to-date.
Digging through my logs (/var/log/libvirt/libvirt.log), I see that I last
successfully booted this LXC container on 2013-07-22, with libvirt
reporting version "1.1.0".
Thoughts?
ostara lxc # equery l libvirt
* Searching for libvirt ...
[IP-] [ ] app-emulation/libvirt-1.1.2-r1:0
ostara lxc # uname -a
Linux ostara 3.10.7-gentoo #2 SMP PREEMPT Sat Aug 24 16:03:57 CDT 2013
x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux
ostara ~ # emerge -pvuND world
These are the packages that would be merged, in order:
Calculating dependencies... done!
Total: 0 packages, Size of downloads: 0 kB
ostara lxc # virsh -c lxc:/// start dwj-hfax-dev
error: Failed to start domain dwj-hfax-dev
error: internal error: guest failed to start: PATH=/bin:/sbin TERM=linux
container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
error receiving signal from container: Input/output error
ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log
2013-09-10 13:56:22.767+0000: starting up
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3
LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log
/usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none
--handshake 23 --background --veth veth1
PATH=/bin:/sbin TERM=linux container=lxc-libvirt
container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
error receiving signal from container: Input/output error
ostara ~ # tail /var/log/libvirt/libvirtd.log
2013-09-10 13:56:22.998+0000: 12948: info : libvirt version: 1.1.2
2013-09-10 13:56:22.998+0000: 12948: error : virLXCControllerRun:2186 :
error receiving signal from container: Input/output error
2013-09-10 13:56:23.053+0000: 12948: error : virCommandWait:2348 : internal
error: Child process (ip link del veth1) unexpected exit status 1: Cannot
find device "veth1"
2013-09-10 13:56:23.097+0000: 10451: error : virNetSocketReadWire:1369 :
Cannot recv data: Connection reset by peer
2013-09-10 13:56:23.153+0000: 10453: error : virLXCProcessStart:1234 :
internal error: guest failed to start: PATH=/bin:/sbin TERM=linux
container=lxc-libvirt container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
error receiving signal from container: Input/output error
2013-09-10 13:56:23.343+0000: 10453: error : virCommandWait:2348 : internal
error: Child process (ip link del veth1) unexpected exit status 1: Cannot
find device "veth1"
(first occurrence of "veth1" in my libvirtd logs)
ostara ~ # grep -a veth1 /var/log/libvirt/libvirtd.log | head -1
2013-09-10 13:51:55.333+0000: 9742: error : virCommandWait:2348 : internal
error: Child process (ip link del veth1) unexpected exit status 1: Cannot
find device "veth1"
ostara ~ # which ip
/bin/ip
ostara ~ # equery b /bin/ip
* Searching for /bin/ip ...
sys-apps/iproute2-3.8.0 (/bin/ip)
ostara ~ # ls -l /bin/ip
-rwxr-xr-x 1 root root 282616 May 4 04:08 /bin/ip
ostara ~ # ifconfig -a | grep "^[a-z]"
br0: flags=4355<UP,BROADCAST,PROMISC,MULTICAST> mtu 1500
br1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
enp6s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev
<domain type='lxc'>
<name>dwj-hfax-dev</name>
<uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>4</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64'>exe</type>
<init>/sbin/init</init>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/libvirt_lxc</emulator>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/vm/lxc/dwj-hfax-dev'/>
<target dir='/'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/usr/portage'/>
<target dir='/usr/portage'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/usr/src'/>
<target dir='/usr/src'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/home'/>
<target dir='/home'/>
</filesystem>
<interface type='bridge'>
<mac address='82:00:00:00:01:01'/>
<source bridge='br0'/>
</interface>
<console type='pty'>
<target type='lxc' port='0'/>
</console>
</devices>
</domain>
1:34 p.m.
On Tue, Sep 10, 2013 at 9:09 AM, Dennis Jenkins
<dennis.jenkins.75(a)gmail.com> wrote:
I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest
available). I
have not used LXC containers for a few weeks, so I don't recall what version
of libvirt I was using when my container last booted successfully.
Unfortunately, Gentoo's portage tree does not offer any previous versions of
libvirt that I could downgrade to.
Yeah our security people got a bit over zealous. That's being rectified.
TL;DR: My container is configured to use "br0" for its networking.
"br0"
exists totally inside my linux server - it is NOT bound to any physical NIC.
"br0" is used for most of my QEMU and LXC VMs. libvirt is reporting that it
cannot find device "veth1". All of my Gentoo packages are up-to-date.
Digging through my logs (/var/log/libvirt/libvirt.log), I see that I last
successfully booted this LXC container on 2013-07-22, with libvirt reporting
version "1.1.0".
Thoughts?
You really need to look at /var/log/libvirt/lxc/dwj-hfax-dev.log
A suggestion would be to make sure that you have all the necessary
kernel options enabled. You can check with: ebuild
/usr/portage/app-emulation/libvirt/libvirt-1.1.2-r1.ebuild setup clean
--
Doug Goldstein
3:44 a.m.
On Tue, Sep 10, 2013 at 09:09:44AM -0500, Dennis Jenkins wrote:
I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest
available). I
have not used LXC containers for a few weeks, so I don't recall what
version of libvirt I was using when my container last booted successfully.
ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log
2013-09-10 13:56:22.767+0000: starting up
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3
LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log
/usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none
--handshake 23 --background --veth veth1
PATH=/bin:/sbin TERM=linux container=lxc-libvirt
container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
error receiving signal from container: Input/output error
We'll need you to increase the log level to get more detail in this
file.
1. Stop the 'libvirtd' daemon
2. Edit /etc/libvirt/lxc.conf and set log_with_libvirtd=1
3. Run
$ LIBVIRT_LOG_FILTERS=1:lxc LIBVIRT_LOG_OUTPUTS=1:stderr /usr/sbin/libvirtd
4. rm -f /var/log/libvirt/lxc/dwj-hfax-dev.log
5. Try to start your guest with virsh
Now send us the much more verbose logfile that was created
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
3:52 a.m.
On 09/11/2013 04:44 PM, Daniel P. Berrange wrote:
On Tue, Sep 10, 2013 at 09:09:44AM -0500, Dennis Jenkins wrote:
> I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest available). I
> have not used LXC containers for a few weeks, so I don't recall what
> version of libvirt I was using when my container last booted successfully.
>
> ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log
> 2013-09-10 13:56:22.767+0000: starting up
>
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3
> LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log
> /usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none
> --handshake 23 --background --veth veth1
> PATH=/bin:/sbin TERM=linux container=lxc-libvirt
> container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
> LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
> LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
> error receiving signal from container: Input/output error
We'll need you to increase the log level to get more detail in this
file.
1. Stop the 'libvirtd' daemon
2. Edit /etc/libvirt/lxc.conf and set log_with_libvirtd=1
3. Run
$ LIBVIRT_LOG_FILTERS=1:lxc LIBVIRT_LOG_OUTPUTS=1:stderr /usr/sbin/libvirtd
4. rm -f /var/log/libvirt/lxc/dwj-hfax-dev.log
5. Try to start your guest with virsh
Now send us the much more verbose logfile that was created
Dennis has already knew what's wrong with his environment,
his kernel doesn't support securityfs. libvirt should handle this problem.
8:41 a.m.
On Wed, Sep 11, 2013 at 3:52 AM, Gao feng <gaofeng(a)cn.fujitsu.com> wrote:
On 09/11/2013 04:44 PM, Daniel P. Berrange wrote:
> On Tue, Sep 10, 2013 at 09:09:44AM -0500, Dennis Jenkins wrote:
>> I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest
available).
I
>> have not used LXC containers for a few weeks, so I don't recall what
>> version of libvirt I was using when my container last booted
successfully.
>
> We'll need you to increase the log level to get more detail in this
> file.
>
> 1. Stop the 'libvirtd' daemon
>
> 2. Edit /etc/libvirt/lxc.conf and set log_with_libvirtd=1
>
> 3. Run
>
> $ LIBVIRT_LOG_FILTERS=1:lxc LIBVIRT_LOG_OUTPUTS=1:stderr
/usr/sbin/libvirtd
>
> 4. rm -f /var/log/libvirt/lxc/dwj-hfax-dev.log
>
> 5. Try to start your guest with virsh
>
>
> Now send us the much more verbose logfile that was created
Dennis has already knew what's wrong with his environment,
his kernel doesn't support securityfs. libvirt should handle this problem.
I already have logs with that level of debug logging:
ostara ~ # grep "log_with_libvirtd" /etc/libvirt/lxc.conf
log_with_libvirtd = 1
If it would be useful, I can revert my kernel, reboot, and run libvirtd
with the extra env vars set, as listed above. Just let me know. :)
I still have the logs from before I upgrades to "libvirt-1.1.2-r1" (lxc
worked for me), and after (libvirt failed). I've had the extra logging
turned on since last spring when you guys helped me then. There is not
much interesting (to me anyway) in the log file:
### last known LXC boot before it broke
2013-07-31 12:49:08.362+0000: starting up
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3
LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log
/usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 24 --security=none
--handshake 27 --background --veth veth1
PATH=/bin:/sbin TERM=linux container=lxc-libvirt
container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
### first attempt to boot LXC after upgrading my kernel and libvirt:
2013-09-10 13:51:54.977+0000: starting up
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3
LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log
/usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 21 --security=none
--handshake 24 --background --veth veth1
PATH=/bin:/sbin TERM=linux container=lxc-libvirt
container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
error receiving signal from container: Input/output error
10:58 a.m.
On Wed, Sep 11, 2013 at 3:52 AM, Gao feng <gaofeng(a)cn.fujitsu.com> wrote:
On 09/11/2013 04:44 PM, Daniel P. Berrange wrote:
> On Tue, Sep 10, 2013 at 09:09:44AM -0500, Dennis Jenkins wrote:
>> I recently upgraded "libvirt" on Gentoo to 1.2.2-r1 (latest available).
I
>> have not used LXC containers for a few weeks, so I don't recall what
>> version of libvirt I was using when my container last booted successfully.
>>
>> ostara ~ # tail /var/log/libvirt/lxc/dwj-hfax-dev.log
>> 2013-09-10 13:56:22.767+0000: starting up
>>
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3:/usr/x86_64-pc-linux-gnu/i686-pc-mingw32/gcc-bin/4.7.3
>> LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:file:/var/log/libvirt/libvirtd.log
>> /usr/libexec/libvirt_lxc --name dwj-hfax-dev --console 20 --security=none
>> --handshake 23 --background --veth veth1
>> PATH=/bin:/sbin TERM=linux container=lxc-libvirt
>> container_uuid=681410de-7b56-41bd-b38d-3c66ce97e7b3
>> LIBVIRT_LXC_UUID=681410de-7b56-41bd-b38d-3c66ce97e7b3
>> LIBVIRT_LXC_NAME=dwj-hfax-dev /sbin/init
>> error receiving signal from container: Input/output error
>
> We'll need you to increase the log level to get more detail in this
> file.
>
> 1. Stop the 'libvirtd' daemon
>
> 2. Edit /etc/libvirt/lxc.conf and set log_with_libvirtd=1
>
> 3. Run
>
> $ LIBVIRT_LOG_FILTERS=1:lxc LIBVIRT_LOG_OUTPUTS=1:stderr /usr/sbin/libvirtd
>
> 4. rm -f /var/log/libvirt/lxc/dwj-hfax-dev.log
>
> 5. Try to start your guest with virsh
>
>
> Now send us the much more verbose logfile that was created
Dennis has already knew what's wrong with his environment,
his kernel doesn't support securityfs. libvirt should handle this problem.
Its 6807238d that made securityfs required on the host kernel when
userns isn't used.
--
Doug Goldstein
4054
days inactive
4055
days old
5 comments
4 participants
participants (4)
-
Daniel P. Berrange -
Dennis Jenkins -
Doug Goldstein -
Gao feng