Re: [libvirt] Loosing lxc guests when restarting libvirt

Hi Cedric,x On Wed, Dec 21, 2016 at 02:36:39PM +0100, Cedric Bosdonnat wrote: I had a short look and it seems like this sequence is killing all running libvirt-lxc guests reliably: # no lxc guest running yet export LIBVIRT_DEFAULT_URI=lxc:/// DOMAIN=sl systemctl daemon-reload # start lxc guest virsh start ${DOMAIN} sleep 1 # give vm some time to start systemctl restart libvirtd virsh list | grep -qs "${DOMAIN}[[:space:]]\+running" # lxc guest gone The important part is the "systemctl daemon-reload". If one leaves that out libvirtd restarts don't kill off any lxc-domains anymore. The issue is that libvirt on reattach fails virCgroupNewDetectMachine due to /proc/<pid-of-lxc-container>/cgroup having changed after libvird's restart: Before systemctl restarts libvirtd: 10:perf_event:/machine/lxc-21383-sl.libvirt-lxc 9:cpuset:/machine/lxc-21383-sl.libvirt-lxc 8:net_cls,net_prio:/machine/lxc-21383-sl.libvirt-lxc 7:pids:/system.slice/libvirtd.service 6:memory:/machine/lxc-21383-sl.libvirt-lxc 5:cpu,cpuacct:/machine/lxc-21383-sl.libvirt-lxc 4:devices:/machine/lxc-21383-sl.libvirt-lxc 3:freezer:/machine/lxc-21383-sl.libvirt-lxc 2:blkio:/machine/lxc-21383-sl.libvirt-lxc 1:name=systemd:/system.slice/libvirtd.service After systemctl restart libvirtd: 10:perf_event:/machine/lxc-21383-sl.libvirt-lxc 9:cpuset:/machine/lxc-21383-sl.libvirt-lxc 8:net_cls,net_prio:/machine/lxc-21383-sl.libvirt-lxc 7:pids:/system.slice/libvirtd.service 6:memory:/system.slice/libvirtd.service 5:cpu,cpuacct:/system.slice/libvirtd.service 4:devices:/system.slice/libvirtd.service 3:freezer:/machine/lxc-21383-sl.libvirt-lxc 2:blkio:/system.slice/libvirtd.service 1:name=systemd:/system.slice/libvirtd.service so the process is moved to other memory, cpu, device and blkio cgroups and therefore libvirtd can't find it anymore. The error in the log looks like: debug : virCgroupValidateMachineGroup:333 : Name 'libvirtd.service' for controller 'cpu' does not match 'sl', 'lxc-21383-sl', 'sl.libvirt-lxc', 'machine-lxc\x2dsl.scope' or 'machine-lxc\x2d21383\x2dsl.scope' This does _not_ happen if one restarts libvirtd right after the "systemctl daemon-reload" or if one drops the "systemctl daemon-reload" from the above example. This also does not happen if one stops libvird via systemd but starts it as /usr/sbin/libvirtd directly. So the culprit happens when * systemctl daemon-reload * libvirtd is restared via systemctl I've looked at audit logs and straced pid 1 without spotting anything. Any ideas where to go looking now? This is systemd 232. Cheers, -- Guido