[libvirt-users] cgroups inside LXC containers losts memory limits after some time
Hi all I have CentOS Linux release 7.0.1406, libvirt 1.2.7 installed. Just after create and start inside LXC container present cgroups. Example for memory: [root@ce7-t1 /]# ls -la /sys/fs/cgroup/memory/ total 0 drwxr-xr-x 2 root root 0 Sep 15 17:14 . drwxr-xr-x 12 root root 280 Sep 15 17:14 .. -rw-r--r-- 1 root root 0 Sep 15 17:14 cgroup.clone_children --w--w--w- 1 root root 0 Sep 15 17:14 cgroup.event_control -rw-r--r-- 1 root root 0 Sep 15 17:15 cgroup.procs -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.failcnt --w------- 1 root root 0 Sep 15 17:14 memory.force_empty -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.kmem.failcnt -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.kmem.limit_in_bytes -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.kmem.max_usage_in_bytes -r--r--r-- 1 root root 0 Sep 15 17:14 memory.kmem.slabinfo -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.kmem.tcp.failcnt -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.kmem.tcp.limit_in_bytes -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.kmem.tcp.max_usage_in_bytes -r--r--r-- 1 root root 0 Sep 15 17:14 memory.kmem.tcp.usage_in_bytes -r--r--r-- 1 root root 0 Sep 15 17:14 memory.kmem.usage_in_bytes -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.limit_in_bytes -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.max_usage_in_bytes -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.memsw.failcnt -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.memsw.limit_in_bytes -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.memsw.max_usage_in_bytes -r--r--r-- 1 root root 0 Sep 15 17:14 memory.memsw.usage_in_bytes -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.move_charge_at_immigrate -r--r--r-- 1 root root 0 Sep 15 17:14 memory.numa_stat -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.oom_control ---------- 1 root root 0 Sep 15 17:14 memory.pressure_level -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.soft_limit_in_bytes -r--r--r-- 1 root root 0 Sep 15 17:14 memory.stat -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.swappiness -r--r--r-- 1 root root 0 Sep 15 17:14 memory.usage_in_bytes -rw-r--r-- 1 root root 0 Sep 15 17:14 memory.use_hierarchy -rw-r--r-- 1 root root 0 Sep 15 17:14 notify_on_release -rw-r--r-- 1 root root 0 Sep 15 17:14 tasks Command "free" inside LXC showed almost normal values: [root@ce7-t1 /]# free total used free shared buffers cached Mem: 1048576 32972 1015604 4473848 0 -4445364 -/+ buffers/cache: 4478336 -3429760 Swap: 1048576 0 1048576 (some problem with negative values) After unpredictable time passed (1-5 day ?), cgroups inside LXC magicaly removed. "free" in such containers show 2^53-1 as maximum values: [root@puppet01 /]# free total used free shared buffers cached Mem: 9007199254740991 591180 9007199254149811 0 0 267924 -/+ buffers/cache: 323256 9007199254417735 Swap: 0 0 0 And no more any cgroups presented at least in memory category: [root@puppet01 /]# ls -la /sys/fs/cgroup/memory/ total 0 b.r. Maxim Kozin
HI all
After unpredictable time passed (1-5 day ?), cgroups inside LXC magicaly removed.
virsh dumpxml config look like this: <domain type='lxc' id='3566'> <name>puppet</name> <uuid>6d49b280-5686-4e3c-b048-1b5d362fb137</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memtune> <hard_limit unit='KiB'>8388608</hard_limit> <soft_limit unit='KiB'>8388608</soft_limit> <swap_hard_limit unit='KiB'>9437184</swap_hard_limit> </memtune> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/libvirt_lxc</emulator> <filesystem type='block' accessmode='passthrough'> <source dev='/dev/data/puppet'/> <target dir='/'/> </filesystem> <interface type='direct'> <mac address='02:00:00:25:a7:56'/> <source dev='br502' mode='bridge'/> <model type='virtio'/> </interface> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='lxc' port='0'/> <alias name='console0'/> </console> </devices> <seclabel type='none'/> </domain> Problem go away after restart LXC container. But occurred again after some time passed. For containers with broken cgroups virsh memtune fails with error: [root@]# virsh memtune --hard-limit 8G puppet --live --config error: Unable to change memory parameters error: unable to set memory hard_limit tunable: Operation not permitted b.r. Maxim Kozin
After unpredictable time passed (1-5 day ?), cgroups inside LXC magicaly removed. I use next workaround for such containers:
1) restore subgroup in cgroup hierarchy : [root@]# mkdir /sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dpuppet.scope 2) put value in limits: [root@]# echo 8589934592 > /sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dpuppet.scope/memory.limit_in_bytes [root@]# echo 9663676416 > /sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dphosgene7.infra.scope/memory.memsw.limit_in_bytes [root@]# echo 8589934592 > /sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dpuppet.scope/memory.kmem.limit_in_bytes [root@]# echo 8589934592 > /sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dpuppet.scope/memory.kmem.tcp.limit_in_bytes [root@]# echo 8589934592 > /sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2dpuppet.scope/memory.soft_limit_in_bytes 3) check that inside container limits still unset: [root@puppet01 /]# free total used free shared buffers cached Mem: 9007199254740991 1591360 9007199253149631 0 0 913112 -/+ buffers/cache: 678248 9007199254062743 Swap: 0 0 0 4) get libvirtd pid of such container: [root@]# ps ax | grep puppet | grep libvirt 3142 ? Ssl 0:17 /usr/libexec/libvirt_lxc --name puppet --console 21 --security=none --handshake 27 --background --veth macvlan0 Moreover, you need get pid of all process existed in LXC. We would test in ssh sessions, because get pid of sshd run in our container. [root@]# pstree -ap 3142 | grep ssh | `-sshd,5527 5) Assign pid with cgroups: [root@]# cgclassify -g memory:machine.slice/machine-lxc\\x2dpuppet.scope 3142 3143 5527 (I put pid 3143 in list - it's init inside LXC) 6) check again in container - start new ssh session: [root@puppet01 /]# free total used free shared buffers cached Mem: 8388608 392 8388216 0 0 0 -/+ buffers/cache: 392 8388216 Swap: 1048576 0 1048576 7) compile and run loadmem: #include <stdlib.h> int main() { int *p; while(1) { int inc=1024*1024*sizeof(char); p=(int*) calloc(1,inc); if(!p) break; } } After 8G allocated process loadmem would be killed b.r. Maxim Kozin
participants (1)
-
mxs kolo