The debug log ends with this: 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:708 : Make group /machine/dwj-hfax-dev.libvirt-lxc 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make controller /sys/fs/cgroup/cpu/machine/dwj-hfax-dev.libvirt-lxc/ 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make controller /sys/fs/cgroup/cpuacct/machine/dwj-hfax-dev.libvirt-lxc/ 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make controller /sys/fs/cgroup/cpuset/machine/dwj-hfax-dev.libvirt-lxc/ 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make controller /sys/fs/cgroup/memory/machine/dwj-hfax-dev.libvirt-lxc/ 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make controller /sys/fs/cgroup/devices/machine/dwj-hfax-dev.libvirt-lxc/ 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make controller /sys/fs/cgroup/freezer/machine/dwj-hfax-dev.libvirt-lxc/ 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make controller /sys/fs/cgroup/blkio/machine/dwj-hfax-dev.libvirt-lxc/ 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:715 : Skipping unmounted controller net_cls 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:729 : Make controller /sys/fs/cgroup/perf_event/machine/dwj-hfax-dev.libvirt-lxc/ 2013-07-12 16:43:31.740+0000: 21365: debug : virCgroupMakeGroup:779 : Done making controllers for group 2013-07-12 16:43:31.740+0000: 21365: debug : virFileMakePathHelper:1995 : path=/var/log/libvirt/lxc mode=0777 2013-07-12 16:43:31.740+0000: 21365: debug : virLXCProcessStart:1096 : Setting current domain def as transient 2013-07-12 16:43:31.741+0000: 21365: debug : virLXCProcessStart:1121 : Preparing host devices 2013-07-12 16:43:31.741+0000: 21365: debug : virLXCProcessStart:1139 : Generating domain security label (if required) ====== end of log ===== Segmentation fault (core dumped) (gdb) bt #0 0x00007fe4750c5d76 in __strcmp_sse42 () from /lib64/libc.so.6 #1 0x00007fe47578ad31 in virSecurityManagerGenLabel () from /usr/lib64/libvirt.so.0 #2 0x00007fe46aa92979 in virLXCProcessStart () from /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so #3 0x00007fe46aa9736e in lxcDomainCreateWithFlags () from /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so #4 0x00007fe47569c067 in virDomainCreate () from /usr/lib64/libvirt.so.0 #5 0x00007fe4760d5578 in remoteDispatchDomainCreateHelper () #6 0x00007fe4756fcd78 in virNetServerProgramDispatch () from /usr/lib64/libvirt.so.0 #7 0x00007fe4756f7302 in virNetServerProcessMsg () from /usr/lib64/libvirt.so.0 #8 0x00007fe4756f7a93 in virNetServerHandleJob () from /usr/lib64/libvirt.so.0 #9 0x00007fe47560f95e in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #10 0x00007fe47560efc6 in virThreadHelper () from /usr/lib64/libvirt.so.0 #11 0x00007fe475354da6 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fe47508d99d in clone () from /lib64/libc.so.6 (gdb) On Fri, Jul 12, 2013 at 11:40 AM, Dennis Jenkins < dennis.jenkins.75(a)gmail.com&gt; wrote: > Hello all, > > I have two issues: > > 1) I am unable to start a seemingly correct LXC domain (I cloned it from > a working domain). > > 2) I am able to crash "libvirtd" by attempting to start the cloned > domain, but starting the original works just fine. > > I humbly submit that item #2 is a bug - the "libvirtd" daemon should > never crash due to anything the "libvirt" client throws at it. As for > item #1, I'm not sure where I went wrong. A full walk-through is below > (ending with a DIFF of the XML from the two domains). > > I created by original domain ("dwj-lnx-dev") a long time ago. Today > I created the new domain ("dwj-hfax-dev") as follows: > > 1) Shutdown "dwj-lnx-dev" > 2) Clone the root file system: "cd /vm/lxc/; cp -a dwj-lnx-dev > dwj-hfax-dev" (2.5GB, ~5 min) > 3) "libvirt -c lxc:/// dumpxml dwj-lnx-dev > a.xml" > 4) ${EDITOR} a.xml > a) changed MAC address, name, memory, source directory for "/" > 5) "libvirt -c lxc:/// define a.xml" > 6) Edit "/etc/bind/pri/*" and "/etc/dhcp/dhcpd.conf" on my host. > > > It does not matter is "dwj-lnx-dev" is running or not. Any attempt > to start "dwj-hfax-dev" will crash libvirtd. > > > In the past I was asked to turn on some debugging and capture a > detailed log ( > https://www.redhat.com/archives/libvirt-users/2013-May/msg00076.html). > I will do this soon and post my results as a follow up. > > > > ostara ~ # uname -a > Linux ostara 3.8.13-gentoo #1 SMP PREEMPT Mon Jun 3 17:10:56 CDT 2013 > x86_64 Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz GenuineIntel GNU/Linux > > > ostara ~ # equery l libvirt > * Searching for libvirt ... > [IP-] [ ] app-emulation/libvirt-1.1.0-r1:0 > > > ostara ~ # virsh -c lxc:/// version > Compiled against library: libvirt 1.1.0 > Using library: libvirt 1.1.0 > Using API: LXC 1.1.0 > Running hypervisor: LXC 3.8.13 > > > ostara ~ # /etc/init.d/libvirtd restart > * Caching service dependencies > ... > [ ok ] > * Stopping libvirtd ... > * Shutting down network(s): > * > default > [ ok ] > * Starting libvirtd > ... > [ ok ] > > > ostara ~ # virsh -c lxc:/// list --all > Id Name State > ---------------------------------------------------- > - dwj-hfax-dev shut off > - dwj-lnx-dev shut off > - vm1 shut off > > ostara ~ # virsh -c lxc:/// start dwj-lnx-dev > Domain dwj-lnx-dev started > > > ostara ~ # virsh -c lxc:/// list --all > Id Name State > ---------------------------------------------------- > 9441 dwj-lnx-dev running > - dwj-hfax-dev shut off > - vm1 shut off > > > ostara ~ # virsh -c lxc:/// start dwj-hfax-dev > error: Failed to start domain dwj-hfax-dev > error: End of file while reading data: Input/output error > error: One or more references were leaked after disconnect from the > hypervisor > error: Failed to reconnect to the hypervisor > > > ostara ~ # virsh -c lxc:/// list --all > error: failed to connect to the hypervisor > error: no valid connection > error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': > Connection refused > > > ostara ~ # ls -l /var/run/libvirt/libvirt-sock > srwx------ 1 root root 0 Jul 12 11:21 /var/run/libvirt/libvirt-sock > > > ostara ~ # ps axfw | grep libvirt > 9997 pts/2 S+ 0:00 \_ grep --colour=auto > libvirt > 8446 ? S 0:00 /usr/sbin/dnsmasq > --conf-file=/var/lib/libvirt/dnsmasq/default.conf > 9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev > --console 19 --security=none --handshake 23 --background --veth veth1 > > > ostara ~ # /etc/init.d/libvirtd restart > * Stopping libvirtd ... > * start-stop-daemon: no matching processes > found > [ ok ] > * Starting libvirtd > ... > [ ok ] > > > ostara ~ # ps axfw | grep libvirt > 10130 pts/2 S+ 0:00 \_ grep --colour=auto > libvirt > 8446 ? S 0:00 /usr/sbin/dnsmasq > --conf-file=/var/lib/libvirt/dnsmasq/default.conf > 9441 ? Ss 0:00 /usr/libexec/libvirt_lxc --name dwj-lnx-dev > --console 19 --security=none --handshake 23 --background --veth veth1 > 10033 ? Sl 0:00 /usr/sbin/libvirtd -d --listen > > > ostara ~ # virsh -c lxc:/// list --all > Id Name State > ---------------------------------------------------- > 9441 dwj-lnx-dev running > - dwj-hfax-dev shut off > - vm1 shut off > > > ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev > <domain type='lxc'> > <name>dwj-hfax-dev</name> > <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid> > <memory unit='KiB'>4194304</memory> > <currentMemory unit='KiB'>4194304</currentMemory> > <vcpu placement='static'>4</vcpu> > <resource> > <partition>/machine</partition> > </resource> > <os> > <type arch='x86_64'>exe</type> > <init>/sbin/init</init> > </os> > <clock offset='utc'/> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>destroy</on_crash> > <devices> > <emulator>/usr/libexec/libvirt_lxc</emulator> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/vm/lxc/dwj-hfax-dev'/> > <target dir='/'/> > </filesystem> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/usr/portage'/> > <target dir='/usr/portage'/> > </filesystem> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/usr/src'/> > <target dir='/usr/src'/> > </filesystem> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/home'/> > <target dir='/home'/> > </filesystem> > <interface type='bridge'> > <mac address='82:00:00:00:01:01'/> > <source bridge='br0'/> > <target dev='veth0'/> > </interface> > <console type='pty'> > <target type='lxc' port='0'/> > </console> > </devices> > <seclabel type='none'/> > </domain> > > > ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev > <domain type='lxc' id='9441'> > <name>dwj-lnx-dev</name> > <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid> > <memory unit='KiB'>500000</memory> > <currentMemory unit='KiB'>500000</currentMemory> > <vcpu placement='static'>2</vcpu> > <resource> > <partition>/machine</partition> > </resource> > <os> > <type arch='x86_64'>exe</type> > <init>/sbin/init</init> > </os> > <clock offset='utc'/> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>destroy</on_crash> > <devices> > <emulator>/usr/libexec/libvirt_lxc</emulator> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/vm/lxc/dwj-lnx-dev'/> > <target dir='/'/> > </filesystem> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/usr/portage'/> > <target dir='/usr/portage'/> > </filesystem> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/usr/src'/> > <target dir='/usr/src'/> > </filesystem> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/home'/> > <target dir='/home'/> > </filesystem> > <interface type='bridge'> > <mac address='82:00:00:00:01:00'/> > <source bridge='br0'/> > <target dev='veth0'/> > </interface> > <console type='pty' tty='/dev/pts/3'> > <source path='/dev/pts/3'/> > <target type='lxc' port='0'/> > <alias name='console0'/> > </console> > </devices> > <seclabel type='none'/> > </domain> > > > ostara ~ # virsh -c lxc:/// dumpxml dwj-lnx-dev > lnx.xml > > > ostara ~ # virsh -c lxc:/// dumpxml dwj-hfax-dev > hfax.xml > > > ostara ~ # diff lnx.xml hfax.xml > 1,6c1,6 > < <domain type='lxc' id='9441'> > < <name>dwj-lnx-dev</name> > < <uuid>fbcd8c3a-9939-12b4-727d-5d3526bc448f</uuid> > < <memory unit='KiB'>500000</memory> > < <currentMemory unit='KiB'>500000</currentMemory> > < <vcpu placement='static'>2</vcpu> > --- > > <domain type='lxc'> > > <name>dwj-hfax-dev</name> > > <uuid>681410de-7b56-41bd-b38d-3c66ce97e7b3</uuid> > > <memory unit='KiB'>4194304</memory> > > <currentMemory unit='KiB'>4194304</currentMemory> > > <vcpu placement='static'>4</vcpu> > 21c21 > < <source dir='/vm/lxc/dwj-lnx-dev'/> > --- > > <source dir='/vm/lxc/dwj-hfax-dev'/> > 37c37 > < <mac address='82:00:00:00:01:00'/> > --- > > <mac address='82:00:00:00:01:01'/> > 41,42c41 > < <console type='pty' tty='/dev/pts/3'> > < <source path='/dev/pts/3'/> > --- > > <console type='pty'> > 44d42 > < <alias name='console0'/> > > > (After reseting everything, and attempting to boot hfax with dev offline, > libvirtd still crashes) > > ostara ~ # virsh -c lxc:/// list --all > Id Name State > ---------------------------------------------------- > - dwj-hfax-dev shut off > - dwj-lnx-dev shut off > - vm1 shut off > > > > ostara ~ # virsh -c lxc:/// start dwj-hfax-dev > error: Failed to start domain dwj-hfax-dev > error: End of file while reading data: Input/output error > error: One or more references were leaked after disconnect from the > hypervisor > error: Failed to reconnect to the hypervisor > >

Interesting. If you are still able to reproduce the crash, can you try to get the line number within virSecurityManagerGenLabel where the crash happened? I think it's the STREQ line (440 linenr). Question is whether model or name is NULL.

wrote:

wrote:

Ah ha! I just learned about "gdb bt full". The existing core dump might have what you need: Line #442. However, the line numbers for the source code in the source tree that my Gentoo system is building from does not match exactly what you listed. Line #442 for me is the one containing the "STREQ" macro: virObjectLock(mgr); for (i = 0; i < vm->nseclabels; i++) { for (j = 0; sec_managers[j]; j++) if (STREQ(vm->seclabels[i]->model, sec_managers[j]->drv->name)) break; I can rebuild with "-O0" and try again. If I can still trigger the crash, the backtrace might have useful values for the optimized variables. I'll post again in a few minutes.