Re: [libvirt] Found mem leak in livirt, need help to debug

W dniu 2015-11-19 o 11:07, Michal Privoznik pisze: > On 18.11.2015 15:33, Piotr Rybicki wrote: >> Hi. >> >> There is a mem leak in libvirt, when doing external snapshot (for backup >> purposes). My KVM domain uses raw storage images via libgfapi. I'm using >> latest 1.2.21 libvirt (although previous versions act the same). >> >> My bash script for snapshot backup uses series of shell commands (virsh >> connect to a remote libvirt host): >> >> * virsh domblklist KVM >> * qemu-img create -f qcow2 -o backing_file=gluster(...) - precreate >> backing file >> * virsh snapshot-create KVM SNAP.xml (...) - create snapshot from >> precreated XML snapshot file >> * cp main img file >> * virsh blockcommit KVM disk (...) >> >> Backup script works fine, however libvirtd process gets bigger and >> bigger each time I run this script. >> >> Some proof of memleak: >> >> 32017 - libvirtd pid >> >> When libvirt started: >> # ps p 32017 o vsz,rss >> VSZ RSS >> 585736 15220 >> >> When I start KVM via virsh start KVM >> # ps p 32017 o vsz,rss >> VSZ RSS >> 1327968 125956 >> >> When i start backup script, after snapshot is created (lots of mem >> allocated) >> # ps p 32017 o vsz,rss >> VSZ RSS >> 3264544 537632 >> >> After backup script finished >> # ps p 32017 o vsz,rss >> VSZ RSS >> 3715920 644940 >> >> When i start backup script for a second time, after snapshot is created >> # ps p 32017 o vsz,rss >> VSZ RSS >> 5521424 1056352 >> >> And so on, until libvirt spills 'Out of memory' when connecting, ane >> being really huge process. >> >> Now, I would like to diagnose it further, to provide detailed >> information about memleak. I tried to use valgrind, but unfortunatelly >> I'm on Opteron 6380 platform, and valgrind doesn't support XOP quitting >> witch SIGILL. >> >> If someone could provide me with detailed information on how to get some >> usefull debug info about this memleak, i'll be more than happy to do it, >> and share results here. > > You can run libvirtd under valgrind (be aware that it will be slow as > snail), then run the reproducer and then just terminate the daemon > (CTRL+C). Valgrind will then report on all the leaks. When doing this I > usually use: > > # valgrind --leak-check=full --show-reachable=yes \ > --child-silent-after-fork=yes libvirtd > > Remember to terminate the system-wide daemon firstly as the one started > under valgrind will die early since you can only have one deamon running > at the time. > > If you are unfamiliar with the output, share it somewhere and I will > take a look. > Thank You Michal. I finally managed to have valgrind running on Opteron 6380. I recomplied with -mno-xop glibc, libvirt and other revelant libs (just for others looking for solution for valgrind on Opteron). Gluster is at 3.5.4 procedure is: start libvirtd start kvm run backup script (with external snapshot) stop kvm stop libvirtd Valgrind output: