Hi Cedric,x On Wed, Dec 21, 2016 at 02:36:39PM +0100, Cedric Bosdonnat wrote:
Hey Christian,
On Tue, 2016-12-20 at 12:29 +0100, Christian Ehrhardt wrote:
Hi, I found an issue in libvirt related to libvirt-lxc, but fail to find the root cause.
The TL;DR is: libvirt-lxc guests get killed on libvirt restart due to "internal error: No valid cgroup for machine"
It was able to reproduce libvirt 1.3.1, 2.4 and 2.5 as packages in Ubuntu and Debian. I wanted to ask for two things: - wider coverage where this does reproduce
I couldn't reproduce here with openSUSE Tumbleweed and libvirt 2.5 packages.
I had a short look and it seems like this sequence is killing all running libvirt-lxc guests reliably: # no lxc guest running yet export LIBVIRT_DEFAULT_URI=lxc:/// DOMAIN=sl systemctl daemon-reload # start lxc guest virsh start ${DOMAIN} sleep 1 # give vm some time to start systemctl restart libvirtd virsh list | grep -qs "${DOMAIN}[[:space:]]\+running" # lxc guest gone The important part is the "systemctl daemon-reload". If one leaves that out libvirtd restarts don't kill off any lxc-domains anymore. The issue is that libvirt on reattach fails virCgroupNewDetectMachine due to /proc/<pid-of-lxc-container>/cgroup having changed after libvird's restart: Before systemctl restarts libvirtd: 10:perf_event:/machine/lxc-21383-sl.libvirt-lxc 9:cpuset:/machine/lxc-21383-sl.libvirt-lxc 8:net_cls,net_prio:/machine/lxc-21383-sl.libvirt-lxc 7:pids:/system.slice/libvirtd.service 6:memory:/machine/lxc-21383-sl.libvirt-lxc 5:cpu,cpuacct:/machine/lxc-21383-sl.libvirt-lxc 4:devices:/machine/lxc-21383-sl.libvirt-lxc 3:freezer:/machine/lxc-21383-sl.libvirt-lxc 2:blkio:/machine/lxc-21383-sl.libvirt-lxc 1:name=systemd:/system.slice/libvirtd.service After systemctl restart libvirtd: 10:perf_event:/machine/lxc-21383-sl.libvirt-lxc 9:cpuset:/machine/lxc-21383-sl.libvirt-lxc 8:net_cls,net_prio:/machine/lxc-21383-sl.libvirt-lxc 7:pids:/system.slice/libvirtd.service 6:memory:/system.slice/libvirtd.service 5:cpu,cpuacct:/system.slice/libvirtd.service 4:devices:/system.slice/libvirtd.service 3:freezer:/machine/lxc-21383-sl.libvirt-lxc 2:blkio:/system.slice/libvirtd.service 1:name=systemd:/system.slice/libvirtd.service so the process is moved to other memory, cpu, device and blkio cgroups and therefore libvirtd can't find it anymore. The error in the log looks like: debug : virCgroupValidateMachineGroup:333 : Name 'libvirtd.service' for controller 'cpu' does not match 'sl', 'lxc-21383-sl', 'sl.libvirt-lxc', 'machine-lxc\x2dsl.scope' or 'machine-lxc\x2d21383\x2dsl.scope' This does _not_ happen if one restarts libvirtd right after the "systemctl daemon-reload" or if one drops the "systemctl daemon-reload" from the above example. This also does not happen if one stops libvird via systemd but starts it as /usr/sbin/libvirtd directly. So the culprit happens when * systemctl daemon-reload * libvirtd is restared via systemctl I've looked at audit logs and straced pid 1 without spotting anything. Any ideas where to go looking now? This is systemd 232. Cheers, -- Guido
- your expertise on the case itself.
It seems that you'll need to check what's going on in virCgroupDetect().
Steps to reproduce: 1. Spawn new KVM Guest of your choice 2. install test dependencies $ apt-get install libvirt-daemon-system libvirt-clients libxml2-utils # or package managers / package names of your chosen os 3. run the following sequence as root export LIBVIRT_DEFAULT_URI=lxc:/// cat << EOF > /tmp/smoke-lxc.xml <domain type='lxc'> <name>sl</name> <memory unit='KiB'>256000</memory> <currentMemory unit='KiB'>256000</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type>exe</type> <init>/bin/bash</init> </os> <features> <privnet/> </features> <clock offset='utc'/> <devices> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
The emulator should be removed from the config for portability purpose: the libvirt_lxc path may vary from a distro / arch to another and libvirt's lxc driver is able to auto-add it.
-- Cedric
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list