Re: [libvirt-users] /proc/meminfo

26 Apr 2016

      Now reporduced with 100%
1) create contrainer with memory limit 1Gb
2) run inside simple memory test allocator:
#include <malloc.h>
#include <unistd.h>
#include <memory.h>
#define MB 1024 * 1024
int main() {
  int total = 0;
  while (1) {
    void *p = malloc( 100*MB );
    memset(p,0, 100*MB );
    total = total + 100;
    printf("Alloc %d Mb\n",total);
    sleep(1);
  }
}
[root@tst-mxs2 ~]# free
              total        used        free      shared  buff/cache   available
Mem:        1048576        7412     1028644       11112       12520     1028644
Swap:       1048576           0     1048576
[root@tst-mxs2 ~]# ./a.out
Alloc 100 Mb
Alloc 200 Mb
Alloc 300 Mb
Alloc 400 Mb
Alloc 500 Mb
Alloc 600 Mb
Alloc 700 Mb
Alloc 800 Mb
Alloc 900 Mb
Alloc 1000 Mb
Killed

As You can see, limit worked and "free" inside container show correct values

3) Check situation outside container, from top hadrware  node:
[root@node01]# cat
/sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2d7445\\x2dtst\\x2dmxs2.test.scope/memory.limit_in_bytes
1073741824
4) Check list of pid in cgroups (it's IMPOTANT moment):
[root@node01]# cat
/sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2d7445\\x2dtst\\x2dmxs2.test.scope/tasks
7445
7446
7480
7506
7510
7511
7512
7529
7532
7533
7723
7724
8251
8253
10455

First PID 7445 - it's pid of  libvirt process for container:
# ps ax | grep 7445
 7445 ?        Sl     0:00 /usr/libexec/libvirt_lxc --name
tst-mxs2.test --console 21 --security=none --handshake 24 --veth
macvlan5
[root@node01]# virsh list
 Id    Name                           State
----------------------------------------------------
 7445  tst-mxs2.test                running

5) Now broke /proc/meminfo inside container.
prepare simple systemd service:
# cat   /usr/lib/systemd/system/true.service
[Unit]
Description=simple test

[Service]
Type=simple
ExecStart=/bin/true

[Install]
WantedBy=multi-user.target

Enable service first time, disable and start:

[root@node01]# systemctl enable   /usr/lib/systemd/system/true.service
Created symlink from
/etc/systemd/system/multi-user.target.wants/true.service to
/usr/lib/systemd/system/true.service.
[root@node01]# systemctl disable true.service
Removed symlink /etc/systemd/system/multi-user.target.wants/true.service.
[root@node01]# systemctl start true.service

Now check memory inside container:
[root@tst-mxs2 ~]# free
              total        used        free      shared  buff/cache   available
Mem:    9007199254740991      190824 9007199254236179       11112
313988 9007199254236179
Swap:             0

6) Check tasks list in cgroups:
[root@node01]# cat
/sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2d7445\\x2dtst\\x2dmxs2.test.scope/tasks
7446
7480
7506
7510
7511
7512
7529
7532
7533
7723
7724
8251
8253

After start disabled systemd service, from task list removed libvirt PID 7445.
It's mean that inside LXC limit real still worked, 7446 - it's PID of
/sbin/init inside container.
Check that limit work:

[root@tst-mxs2 ~]# free
              total        used        free      shared  buff/cache   available
Mem:    9007199254740991      190824 9007199254236179       11112
313988 9007199254236179
Swap:             0           0           0
[root@tst-mxs2 ~]# ./a.out
Alloc 100 Mb
Alloc 200 Mb
Alloc 300 Mb
Alloc 400 Mb
Alloc 500 Mb
Alloc 600 Mb
Alloc 700 Mb
Alloc 800 Mb
Alloc 900 Mb
Alloc 1000 Mb
Killed

Broken only fuse mount.  It's positive news - process inside container
even in case 8Ptb can't allocate more memory that set in cgroups.
But negative news - that some java based sotfware (as puppetdb in our
case)  plan self strategy based on 8Ptb memory and collapsed after
reach real limit.

resume:
 1) don't start disabled service by systemd
 2) workaround by cglassify or by it's simple analog
[root@node01]# echo 7445 >
/sys/fs/cgroup/memory/machine.slice/machine-lxc\\x2d7445\\x2dtst\\x2dmxs2.test.scope/tasks

p.s.
 I am not sure whose  bug - libvirtd or systemd.

b.r.
 Maxim Kozin

On 3/23/16, Peter Steele <pwsteele@gmail.com> wrote:
...
Has anyone seen this issue? We're running containers under CentOS 7.2
and some of these containers are reporting incorrect memory allocation
in /proc/meminfo. The output below comes from a system with 32G of
memory and 84GB of swap. The values reported are completely wrong.
# cat /proc/meminfo
MemTotal:       9007199254740991 kB
MemFree:        9007199224543267 kB
MemAvailable:   12985680 kB
Buffers:               0 kB
Cached:           119744 kB
SwapCached:        10804 kB
Active:           110228 kB
Inactive:         111716 kB
Active(anon):      53840 kB
Inactive(anon):    57568 kB
Active(file):      56388 kB
Inactive(file):    54148 kB
Unevictable:           0 kB
Mlocked:        15347728 kB
SwapTotal:             0 kB
SwapFree:       18446744073709524600 kB
Dirty:             20304 kB
Writeback:         99596 kB
AnonPages:      18963368 kB
Mapped:           231472 kB
Shmem:             51852 kB
Slab:            1891324 kB
SReclaimable:    1805244 kB
SUnreclaim:        86080 kB
KernelStack:       60656 kB
PageTables:        81948 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    104487760 kB
Committed_AS:   31507444 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      354796 kB
VmallocChunk:   34359380456 kAnonHugePages:  15630336 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       81684 kB
DirectMap2M:     3031040 kB
DirectMap1G:    32505856 kB
_______________________________________________
libvirt-users mailing list
libvirt-users@redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users

Re: [libvirt-users] /proc/meminfo

mxs kolo