[libvirt-users] LXC + passthrough mount and host filesystem-cache
Hello, I'm using libvirt to build/run LXC instances. My LXC instances use passthrough filesystem mounts. When I try to do large file systems operations (ex tar or rsync) the file systems cache on the host spikes and causes the OOM handler to run and kills processes in the LXC. Has anyone else seen this? Is there a way around this? At this point I'm resorting to running a cron job that dumps the filesystem cache every 5 minutes. The result is the filesystem cache on the host never grows too large and OOM never runs against LXC processes. The obvious down fall is that I'm killing my filesystem performance by duming the cache. I'm currently running libvirt 1.0.0 -- James R. Leu
On Wed, Mar 26, 2014 at 02:52:58PM -0500, James R. Leu wrote:
Hello,
I'm using libvirt to build/run LXC instances. My LXC instances use passthrough filesystem mounts. When I try to do large file systems operations (ex tar or rsync) the file systems cache on the host spikes and causes the OOM handler to run and kills processes in the LXC.
So it is specifically targetting processes inside the container and not the host ? If you run the same tar/rsync operations outside the container on the same filesystem, presumably you don't see the OOM killer behaviour ?
Has anyone else seen this? Is there a way around this? At this point I'm resorting to running a cron job that dumps the filesystem cache every 5 minutes. The result is the filesystem cache on the host never grows too large and OOM never runs against LXC processes. The obvious down fall is that I'm killing my filesystem performance by duming the cache.
This is the first I've heard of this problem and it certainly seems odd/bad. I wonder if there's some cgroup tunables that are being set badly, or that need to be set ? Or some global proc/sysfs settings Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Daniel, please see my response inline. On Thu, Mar 27, 2014 at 10:48:17AM +0000, Daniel P. Berrange wrote:
On Wed, Mar 26, 2014 at 02:52:58PM -0500, James R. Leu wrote:
Hello,
I'm using libvirt to build/run LXC instances. My LXC instances use passthrough filesystem mounts. When I try to do large file systems operations (ex tar or rsync) the file systems cache on the host spikes and causes the OOM handler to run and kills processes in the LXC.
So it is specifically targetting processes inside the container and not the host ?
If you run the same tar/rsync operations outside the container on the same filesystem, presumably you don't see the OOM killer behaviour ?
Correct. For example, the LXC has <memory unit='KiB'>1048576</memory> The processes inside the LXC are not using anywhere near the the limit, but they are growing (ie doing sbrk to allocate new memory). The rsync or tar are running inside the LXC. If they cause the page cache to grow by more then the LXC memory limit then the next time a process inside the LXC tries to allocate even a small amount of memory, the OOM killer kills that process. It seems like the page cache growth incured by the rsync or tar in the LXC is being attributed to the LXC memory usage, but the page cache is not being freed infavor of memory allocation by processes in the LXC. Is there anything I can do to provide/gather more information?
Has anyone else seen this? Is there a way around this? At this point I'm resorting to running a cron job that dumps the filesystem cache every 5 minutes. The result is the filesystem cache on the host never grows too large and OOM never runs against LXC processes. The obvious down fall is that I'm killing my filesystem performance by duming the cache.
This is the first I've heard of this problem and it certainly seems odd/bad. I wonder if there's some cgroup tunables that are being set badly, or that need to be set ? Or some global proc/sysfs settings
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- James R. Leu | Director of Technology | INOC | Madison, WI, USA O: +1-608-204-0203 | F: +1-608-663-4558 | jleu@inoc.com | www.inoc.com Service. Not Software.®
participants (2)
-
Daniel P. Berrange -
James R. Leu