9:28 a.m.
Hi all,
When libvirt is running on a system with limited number of processes allowed
to be created by the user under which qemu processes are run, we hit
scalability issues.
My take on this issue is that it's in fact a host configuration issue and it
should just be documented that users need to increase the limit if they're
going to run large number of domains or on a large number of VCPUs.
I don't think libvirt should be involved in changing the limit in any way,
especially since it's not possible to change the limit in runtime as doing so
can only affect new processes.
However, I'd like to be sure this is also the thinking of broader community,
esp. Dan :-)
Jirka
6:07 a.m.
On Wed, Mar 02, 2011 at 04:28:46PM +0100, Jiri Denemark wrote:
Hi all,
When libvirt is running on a system with limited number of processes allowed
to be created by the user under which qemu processes are run, we hit
scalability issues.
My take on this issue is that it's in fact a host configuration issue and it
should just be documented that users need to increase the limit if they're
going to run large number of domains or on a large number of VCPUs.
I don't think libvirt should be involved in changing the limit in any way,
especially since it's not possible to change the limit in runtime as doing so
can only affect new processes.
If it were possible to change it on the fly, then I think it could be
valid to suggest it is libvirt's responsibility. Since ulimits can't be
changed for the existing QEMU processes though, the only option is to
change it ahead of time at a host level. The default of 1024 is clearly
faaar to low for modern systems so, IMHO, someone/thing just needs to
place a file at /etc/limits.d/qemu.conf to raise it say to 10,000 which
a modern Linux system can easily cope with.
The original rational for the nproc ulimit is to protect against fork
bombs, but this is really a terrible solution because it applies at the
wrong level. To protect against this properly we would want a limit on
a number of children any single QEMU process can spawn, not a limit on
the number that the QEMU user can spawn. The obvious places for this is
a cgroup tunable, which would mesh nicely with the fact that we put each
QEMU in a dedicated group. Someone has written such a patch before, but
it was never merged AFAICT:
http://lwn.net/Articles/285337/
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
7:46 a.m.
On Thu, Mar 03, 2011 at 12:07:12 +0000, Daniel P. Berrange wrote:
If it were possible to change it on the fly, then I think it could
be
valid to suggest it is libvirt's responsibility. Since ulimits can't be
changed for the existing QEMU processes though, the only option is to
change it ahead of time at a host level. The default of 1024 is clearly
faaar to low for modern systems so, IMHO, someone/thing just needs to
place a file at /etc/limits.d/qemu.conf to raise it say to 10,000 which
a modern Linux system can easily cope with.
Right, although it's /etc/security/limits.d/..., but the thing is who/what
should provided that file. Should it be libvirt or host admin or something
else? On one hand, I think it shouldn't be libvirt since it is possible to
change qemu user in /etc/libvirt/qemu.conf and setting such limits should be
done by the host admin. On the other hand, for better out-of-the-box behavior,
we could generate that file according to how libvirt was configured (i.e.,
what the default qemu user is) and install it.
Which of the options do you prefer?
The original rational for the nproc ulimit is to protect against
fork
bombs, but this is really a terrible solution because it applies at the
wrong level. To protect against this properly we would want a limit on
a number of children any single QEMU process can spawn, not a limit on
the number that the QEMU user can spawn. The obvious places for this is
a cgroup tunable, which would mesh nicely with the fact that we put each
QEMU in a dedicated group. Someone has written such a patch before, but
it was never merged AFAICT:
Yeah, that would ideal for limiting number of process/threads a single domain
can use.
Jirka
5014
days inactive
5015
days old
2 comments
2 participants
participants (2)
-
Daniel P. Berrange -
Jiri Denemark