Hi Cedric,x On Wed, Dec 21, 2016 at 02:36:39PM +0100, Cedric Bosdonnat wrote: I had a short look and it seems like this sequence is killing all running libvirt-lxc guests reliably: # no lxc guest running yet export LIBVIRT_DEFAULT_URI=lxc:/// DOMAIN=sl systemctl daemon-reload # start lxc guest virsh start ${DOMAIN} sleep 1 # give vm some time to start systemctl restart libvirtd virsh list | grep -qs "${DOMAIN}[[:space:]]\+running" # lxc guest gone The important part is the "systemctl daemon-reload". If one leaves that out libvirtd restarts don't kill off any lxc-domains anymore. The issue is that libvirt on reattach fails virCgroupNewDetectMachine due to /proc/<pid-of-lxc-container>/cgroup having changed after libvird's restart: Before systemctl restarts libvirtd: 10:perf_event:/machine/lxc-21383-sl.libvirt-lxc 9:cpuset:/machine/lxc-21383-sl.libvirt-lxc 8:net_cls,net_prio:/machine/lxc-21383-sl.libvirt-lxc 7:pids:/system.slice/libvirtd.service 6:memory:/machine/lxc-21383-sl.libvirt-lxc 5:cpu,cpuacct:/machine/lxc-21383-sl.libvirt-lxc 4:devices:/machine/lxc-21383-sl.libvirt-lxc 3:freezer:/machine/lxc-21383-sl.libvirt-lxc 2:blkio:/machine/lxc-21383-sl.libvirt-lxc 1:name=systemd:/system.slice/libvirtd.service After systemctl restart libvirtd: 10:perf_event:/machine/lxc-21383-sl.libvirt-lxc 9:cpuset:/machine/lxc-21383-sl.libvirt-lxc 8:net_cls,net_prio:/machine/lxc-21383-sl.libvirt-lxc 7:pids:/system.slice/libvirtd.service 6:memory:/system.slice/libvirtd.service 5:cpu,cpuacct:/system.slice/libvirtd.service 4:devices:/system.slice/libvirtd.service 3:freezer:/machine/lxc-21383-sl.libvirt-lxc 2:blkio:/system.slice/libvirtd.service 1:name=systemd:/system.slice/libvirtd.service so the process is moved to other memory, cpu, device and blkio cgroups and therefore libvirtd can't find it anymore. The error in the log looks like: debug : virCgroupValidateMachineGroup:333 : Name 'libvirtd.service' for controller 'cpu' does not match 'sl', 'lxc-21383-sl', 'sl.libvirt-lxc', 'machine-lxc\x2dsl.scope' or 'machine-lxc\x2d21383\x2dsl.scope' This does _not_ happen if one restarts libvirtd right after the "systemctl daemon-reload" or if one drops the "systemctl daemon-reload" from the above example. This also does not happen if one stops libvird via systemd but starts it as /usr/sbin/libvirtd directly. So the culprit happens when * systemctl daemon-reload * libvirtd is restared via systemctl I've looked at audit logs and straced pid 1 without spotting anything. Any ideas where to go looking now? This is systemd 232. Cheers, -- Guido

Hi Guido, On Sun, 2016-12-25 at 00:21 +0100, Guido Günther wrote: Thanks for your work: it really helped me reproduce it here too. After quite a lot of fiddling with the thing I discovered that it happens to me only if daemon-reload is done between the container start and the libvirtd restart. I've also seen that the following doesn't lead to the problem: virsh start sl systemctl daemon-reload systemctl stop libvirtd libvirtd (manual start) But after that, if I kill libvirtd and start it using systemctl start libvirtd, then the container disappears too. I tried debugging this, but didn't come to anything interesting thus far. I'll try again later with a less confused brain ;) -- Cedric