Re: [libvirt-users] libvrtd-1.1.0 crashes when attempting to start some (but not all) LXC containers

Hello all, I have two issues: 1) I am unable to start a seemingly correct LXC domain (I cloned it from a working domain). 2) I am able to crash "libvirtd" by attempting to start the cloned domain, but starting the original works just fine. I humbly submit that item #2 is a bug - the "libvirtd" daemon should never crash due to anything the "libvirt" client throws at it. As for item #1, I'm not sure where I went wrong. A full walk-through is below (ending with a DIFF of the XML from the two domains). I created by original domain ("dwj-lnx-dev") a long time ago. Today I created the new domain ("dwj-hfax-dev") as follows: 1) Shutdown "dwj-lnx-dev" 2) Clone the root file system: "cd /vm/lxc/; cp -a dwj-lnx-dev dwj-hfax-dev" (2.5GB, ~5 min) 3) "libvirt -c lxc:/// dumpxml dwj-lnx-dev > a.xml" 4) ${EDITOR} a.xml a) changed MAC address, name, memory, source directory for "/" 5) "libvirt -c lxc:/// define a.xml" 6) Edit "/etc/bind/pri/*" and "/etc/dhcp/dhcpd.conf" on my host. It does not matter is "dwj-lnx-dev" is running or not. Any attempt to start "dwj-hfax-dev" will crash libvirtd. In the past I was asked to turn on some debugging and capture a detailed log ( https://www.redhat.com/archives/libvirt-users/2013-May/msg00076.html). I will do this soon and post my results as a follow up. ostara ~ # uname -a Linux ostara 3.8.13-gentoo #1 SMP PREEMPT Mon Jun 3 17:10:56 CDT 2013 x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux ostara ~ # equery l libvirt * Searching for libvirt ... [IP-] [ ] app-emulation/libvirt-1.1.0-r1:0 ostara ~ # virsh -c lxc:/// version Compiled against library: libvirt 1.1.0 Using library: libvirt 1.1.0 Using API: LXC 1.1.0 Running hypervisor: LXC 3.8.13 ostara ~ # /etc/init.d/libvirtd restart * Caching service dependencies ... [ ok ] * Stopping libvirtd ... * Shutting down network(s): * default [ ok ] * Starting libvirtd ... [ ok ] ostara ~ # virsh -c lxc:/// list --all Id Name State ---------------------------------------------------- - dwj-hfax-dev shut off - dwj-lnx-dev shut off - vm1 shut off ostara ~ # virsh -c lxc:/// start dwj-lnx-dev Domain dwj-lnx-dev started ostara ~ # virsh -c lxc:/// list --all Id Name State ---------------------------------------------------- 9441 dwj-lnx-dev running - dwj-hfax-dev shut off - vm1 shut off ostara ~ # virsh -c lxc:/// start dwj-hfax-dev error: Failed to start domain dwj-hfax-dev error: End of file while reading data: Input/output error error: One or more references were leaked after disconnect from the hypervisor error: Failed to reconnect to the hypervisor ostara ~ # virsh -c lxc:/// list --all error: failed to connect to the hypervisor error: no valid connection error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused ostara ~ # ls -l /var/run/libvirt/libvirt-sock srwx------ 1 root root 0 Jul 12 11:21 /var/run/libvirt/libvirt-sock ostara ~ # ps axfw | grep libvirt 9997 pts/2 S+ 0:00 \_ grep --colour=auto libvirt 8446 ? S 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf 9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev --console 19 --security=none --handshake 23 --background --veth veth1 ostara ~ # /etc/init.d/libvirtd restart * Stopping libvirtd ... * start-stop-daemon: no matching processes found [ ok ] * Starting libvirtd ... [ ok ] ostara ~ # ps axfw | grep libvirt 10130 pts/2 S+ 0:00 \_ grep --colour=auto libvirt 8446 ? S 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf 9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev --console 19 --security=none --handshake 23 --background --veth veth1 10033 ? Sl 0:00 /usr/sbin/libvirtd -d --listen ostara ~ # virsh -c lxc:/// list --all Id Name State ---------------------------------------------------- 9441 dwj-lnx-dev running - dwj-hfax-dev shut off - vm1 shut off ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev <domain type='lxc'> <name>dwj-hfax-dev</name> <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/vm/lxc/dwj-hfax-dev'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/usr/portage'/> <target dir='/usr/portage'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/usr/src'/> <target dir='/usr/src'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/home'/> <target dir='/home'/> </filesystem> <interface type='bridge'> <mac address='82:00:00:00:01:01'/> <source bridge='br0'/> <target dev='veth0'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> </devices> <seclabel type='none'/> </domain> ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev <domain type='lxc' id='9441'> <name>dwj-lnx-dev</name> <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid> <memory unit='KiB'>500000</memory> <currentMemory unit='KiB'>500000</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/vm/lxc/dwj-lnx-dev'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/usr/portage'/> <target dir='/usr/portage'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/usr/src'/> <target dir='/usr/src'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/home'/> <target dir='/home'/> </filesystem> <interface type='bridge'> <mac address='82:00:00:00:01:00'/> <source bridge='br0'/> <target dev='veth0'/> </interface> <console type='pty' tty='/dev/pts/3'> <source path='/dev/pts/3'/> <target type='lxc' port='0'/> <alias name='console0'/> </console> </devices> <seclabel type='none'/> </domain> ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev > lnx.xml ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev > hfax.xml ostara ~ # diff lnx.xml hfax.xml 1,6c1,6 < <domain type='lxc' id='9441'> < <name>dwj-lnx-dev</name> < <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid> < <memory unit='KiB'>500000</memory> < <currentMemory unit='KiB'>500000</currentMemory> < <vcpu placement='static'>2</vcpu> --- > <domain type='lxc'> > <name>dwj-hfax-dev</name> > <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid> > <memory unit='KiB'>4194304</memory> > <currentMemory unit='KiB'>4194304</currentMemory> > <vcpu placement='static'>4</vcpu> 21c21 < <source dir='/vm/lxc/dwj-lnx-dev'/> --- > <source dir='/vm/lxc/dwj-hfax-dev'/> 37c37 < <mac address='82:00:00:00:01:00'/> --- > <mac address='82:00:00:00:01:01'/> 41,42c41 < <console type='pty' tty='/dev/pts/3'> < <source path='/dev/pts/3'/> --- > <console type='pty'> 44d42 < <alias name='console0'/> (After reseting everything, and attempting to boot hfax with dev offline, libvirtd still crashes) ostara ~ # virsh -c lxc:/// list --all Id Name State ---------------------------------------------------- - dwj-hfax-dev shut off - dwj-lnx-dev shut off - vm1 shut off ostara ~ # virsh -c lxc:/// start dwj-hfax-dev error: Failed to start domain dwj-hfax-dev error: End of file while reading data: Input/output error error: One or more references were leaked after disconnect from the hypervisor error: Failed to reconnect to the hypervisor