[libvirt] [PATCH] daemon: Allow overriding NOFILES ulimit for the daemon as well

One of my latest patches (d8db0f9690) created support for setting the limit for the maximum of opened files by qemu user. However, since libvirtd keeps one FD opened per domain (well, for qemu at least) it will likely hit this limit on huge scenarios. --- daemon/libvirtd.aug | 1 + daemon/libvirtd.c | 37 +++++++++++++++++++++++++++++++++++++ daemon/libvirtd.conf | 9 +++++++++ 3 files changed, 47 insertions(+), 0 deletions(-) diff --git a/daemon/libvirtd.aug b/daemon/libvirtd.aug index 9d78bd7..278099b 100644 --- a/daemon/libvirtd.aug +++ b/daemon/libvirtd.aug @@ -58,6 +58,7 @@ module Libvirtd = | int_entry "max_requests" | int_entry "max_client_requests" | int_entry "prio_workers" + | int_entry "max_files" let logging_entry = int_entry "log_level" | str_entry "log_filters" diff --git a/daemon/libvirtd.c b/daemon/libvirtd.c index b1b542b..34bdd39 100644 --- a/daemon/libvirtd.c +++ b/daemon/libvirtd.c @@ -139,6 +139,8 @@ struct daemonConfig { int max_requests; int max_client_requests; + int max_files; + int log_level; char *log_filters; char *log_outputs; @@ -1081,6 +1083,8 @@ daemonConfigLoad(struct daemonConfig *data, GET_CONF_INT (conf, filename, max_requests); GET_CONF_INT (conf, filename, max_client_requests); + GET_CONF_INT (conf, filename, max_files); + GET_CONF_INT (conf, filename, audit_level); GET_CONF_INT (conf, filename, audit_logging); @@ -1198,6 +1202,34 @@ static int daemonStateInit(virNetServerPtr srv) return 0; } +/* + * daemonSetProcessLimits: + * @config: where to take limits from + * + * Set both soft and hard limits set in config file. + * + * Returns 0 on success, -1 otherwise. + */ +static int +daemonSetProcessLimits(struct daemonConfig *config) +{ + struct rlimit rlim; + + if (config->max_files > 0) { + /* Max number of opened files is one greater than + * actual limit. See man setrlimit */ + rlim.rlim_cur = rlim.rlim_max = config->max_files + 1; + if (setrlimit(RLIMIT_NOFILE, &rlim) < 0) { + virReportSystemError(errno, + _("cannot set max opened files to %d"), + config->max_files); + return -1; + } + } + + return 0; +} + /* Print command-line usage. */ static void daemonUsage(const char *argv0, bool privileged) @@ -1429,6 +1461,11 @@ int main(int argc, char **argv) { exit(EXIT_FAILURE); } + if (daemonSetProcessLimits(config) < 0) { + /* Helper reported error */ + exit(EXIT_FAILURE); + } + if (godaemon) { char ebuf[1024]; diff --git a/daemon/libvirtd.conf b/daemon/libvirtd.conf index 3eab2be..70d5294 100644 --- a/daemon/libvirtd.conf +++ b/daemon/libvirtd.conf @@ -279,6 +279,15 @@ # and max_workers parameter #max_client_requests = 5 +# If max_files is set to positive integer, the maximum number +# of opened files for the daemon is set to given value. This +# can be used to override host system default value. +# Please keep in mind that in some drivers (like qemu) this +# will affect the upper bound of domains which can run simultaneously. +# That's because the daemon keeps one file opened per domain - monitor. +# Well, at least one. +#max_files = 0 + ################################################################# # # Logging controls -- 1.7.3.4

On Thu, Jan 05, 2012 at 11:16:32AM +0100, Michal Privoznik wrote:
One of my latest patches (d8db0f9690) created support for setting the limit for the maximum of opened files by qemu user. However, since libvirtd keeps one FD opened per domain (well, for qemu at least) it will likely hit this limit on huge scenarios. --- daemon/libvirtd.aug | 1 + daemon/libvirtd.c | 37 +++++++++++++++++++++++++++++++++++++ daemon/libvirtd.conf | 9 +++++++++ 3 files changed, 47 insertions(+), 0 deletions(-)
IMHO, this is the kind of thing that should be done by the system init scripts. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On 05.01.2012 11:50, Daniel P. Berrange wrote:
On Thu, Jan 05, 2012 at 11:16:32AM +0100, Michal Privoznik wrote:
One of my latest patches (d8db0f9690) created support for setting the limit for the maximum of opened files by qemu user. However, since libvirtd keeps one FD opened per domain (well, for qemu at least) it will likely hit this limit on huge scenarios. --- daemon/libvirtd.aug | 1 + daemon/libvirtd.c | 37 +++++++++++++++++++++++++++++++++++++ daemon/libvirtd.conf | 9 +++++++++ 3 files changed, 47 insertions(+), 0 deletions(-)
IMHO, this is the kind of thing that should be done by the system init scripts.
Daniel
In that case, can you give me a hint how to change that limit on RHEL-6? Esp. when I am not running libvirtd service by hand but via Upstart init? Because all my attempts to change /etc/security/limits.conf were not successful. Thanks Michal

On Thu, Jan 05, 2012 at 12:23:14PM +0100, Michal Privoznik wrote:
On 05.01.2012 11:50, Daniel P. Berrange wrote:
On Thu, Jan 05, 2012 at 11:16:32AM +0100, Michal Privoznik wrote:
One of my latest patches (d8db0f9690) created support for setting the limit for the maximum of opened files by qemu user. However, since libvirtd keeps one FD opened per domain (well, for qemu at least) it will likely hit this limit on huge scenarios. --- daemon/libvirtd.aug | 1 + daemon/libvirtd.c | 37 +++++++++++++++++++++++++++++++++++++ daemon/libvirtd.conf | 9 +++++++++ 3 files changed, 47 insertions(+), 0 deletions(-)
IMHO, this is the kind of thing that should be done by the system init scripts.
Daniel
In that case, can you give me a hint how to change that limit on RHEL-6? Esp. when I am not running libvirtd service by hand but via Upstart init? Because all my attempts to change /etc/security/limits.conf were not successful.
For SysV init, the /etc/init.d/libvirtd and /etc/sysconfig/libvirt scripts will need to explicitly call ulimit to set this. For SystemD, you can just add LimitNOFILE=NNNN to the libvirtd.service definition Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

One of my latest patches (d8db0f9690) created support for setting the limit for the maximum of opened files by qemu user. However, since libvirtd keeps one FD opened per domain (well, for qemu at least) it will likely hit this limit on huge scenarios. --- daemon/libvirtd.init.in | 7 +++++++ daemon/libvirtd.service.in | 2 ++ daemon/libvirtd.sysconf | 3 +++ daemon/libvirtd.upstart | 5 +++++ 4 files changed, 17 insertions(+), 0 deletions(-) diff --git a/daemon/libvirtd.init.in b/daemon/libvirtd.init.in index 4e610cb..3c49b1f 100644 --- a/daemon/libvirtd.init.in +++ b/daemon/libvirtd.init.in @@ -60,6 +60,13 @@ start() { echo -n $"Starting $SERVICE daemon: " mkdir -p @localstatedir@/cache/libvirt rm -rf @localstatedir@/cache/libvirt/* + + # LIBVIRTD_NOFILES_LIMIT from /etc/sysconfig/libvirtd is not handled + # automatically + if [ -n "$LIBVIRTD_NOFILES_LIMIT" ]; then + ulimit -n "$LIBVIRTD_NOFILES_LIMIT" + fi + KRB5_KTNAME=$KRB5_KTNAME daemon --pidfile $PIDFILE --check $SERVICE $PROCESS --daemon $LIBVIRTD_CONFIG_ARGS $LIBVIRTD_ARGS RETVAL=$? echo diff --git a/daemon/libvirtd.service.in b/daemon/libvirtd.service.in index 9661428..8f2458a 100644 --- a/daemon/libvirtd.service.in +++ b/daemon/libvirtd.service.in @@ -15,6 +15,8 @@ Before=libvirt-guests.service EnvironmentFile=-/etc/sysconfig/libvirtd ExecStart=@sbindir@/libvirtd $LIBVIRTD_ARGS ExecReload=/bin/kill -HUP $MAINPID +# Override the maximum number of opened files +#LimitNOFILE=2048 [Install] WantedBy=multi-user.target diff --git a/daemon/libvirtd.sysconf b/daemon/libvirtd.sysconf index ab273c8..3af1f03 100644 --- a/daemon/libvirtd.sysconf +++ b/daemon/libvirtd.sysconf @@ -19,3 +19,6 @@ #QEMU_AUDIO_DRV=sdl # #SDL_AUDIODRIVER=pulse + +# Override the maximum number of opened files +#LIBVIRTD_NOFILES_LIMIT=2048 diff --git a/daemon/libvirtd.upstart b/daemon/libvirtd.upstart index f51701a..e620f41 100644 --- a/daemon/libvirtd.upstart +++ b/daemon/libvirtd.upstart @@ -31,6 +31,11 @@ script ulimit -c "$DAEMON_COREFILE_LIMIT" fi + # LIBVIRTD_NOFILES_LIMIT from /etc/sysconfig/libvirtd is not handled + # automatically + if [ -n "$LIBVIRTD_NOFILES_LIMIT" ]; then + ulimit -n "$LIBVIRTD_NOFILES_LIMIT" + fi mkdir -p /var/cache/libvirt rm -rf /var/cache/libvirt/* -- 1.7.3.4

On 12.01.2012 18:52, Michal Privoznik wrote:
One of my latest patches (d8db0f9690) created support for setting the limit for the maximum of opened files by qemu user. However, since libvirtd keeps one FD opened per domain (well, for qemu at least) it will likely hit this limit on huge scenarios. --- daemon/libvirtd.init.in | 7 +++++++ daemon/libvirtd.service.in | 2 ++ daemon/libvirtd.sysconf | 3 +++ daemon/libvirtd.upstart | 5 +++++ 4 files changed, 17 insertions(+), 0 deletions(-)
Ping?

On Thu, Jan 12, 2012 at 06:52:52PM +0100, Michal Privoznik wrote:
One of my latest patches (d8db0f9690) created support for setting the limit for the maximum of opened files by qemu user. However, since libvirtd keeps one FD opened per domain (well, for qemu at least) it will likely hit this limit on huge scenarios. --- daemon/libvirtd.init.in | 7 +++++++ daemon/libvirtd.service.in | 2 ++ daemon/libvirtd.sysconf | 3 +++ daemon/libvirtd.upstart | 5 +++++ 4 files changed, 17 insertions(+), 0 deletions(-)
diff --git a/daemon/libvirtd.init.in b/daemon/libvirtd.init.in index 4e610cb..3c49b1f 100644 --- a/daemon/libvirtd.init.in +++ b/daemon/libvirtd.init.in @@ -60,6 +60,13 @@ start() { echo -n $"Starting $SERVICE daemon: " mkdir -p @localstatedir@/cache/libvirt rm -rf @localstatedir@/cache/libvirt/* + + # LIBVIRTD_NOFILES_LIMIT from /etc/sysconfig/libvirtd is not handled + # automatically + if [ -n "$LIBVIRTD_NOFILES_LIMIT" ]; then + ulimit -n "$LIBVIRTD_NOFILES_LIMIT" + fi + KRB5_KTNAME=$KRB5_KTNAME daemon --pidfile $PIDFILE --check $SERVICE $PROCESS --daemon $LIBVIRTD_CONFIG_ARGS $LIBVIRTD_ARGS RETVAL=$? echo diff --git a/daemon/libvirtd.service.in b/daemon/libvirtd.service.in index 9661428..8f2458a 100644 --- a/daemon/libvirtd.service.in +++ b/daemon/libvirtd.service.in @@ -15,6 +15,8 @@ Before=libvirt-guests.service EnvironmentFile=-/etc/sysconfig/libvirtd ExecStart=@sbindir@/libvirtd $LIBVIRTD_ARGS ExecReload=/bin/kill -HUP $MAINPID +# Override the maximum number of opened files +#LimitNOFILE=2048
[Install] WantedBy=multi-user.target diff --git a/daemon/libvirtd.sysconf b/daemon/libvirtd.sysconf index ab273c8..3af1f03 100644 --- a/daemon/libvirtd.sysconf +++ b/daemon/libvirtd.sysconf @@ -19,3 +19,6 @@ #QEMU_AUDIO_DRV=sdl # #SDL_AUDIODRIVER=pulse + +# Override the maximum number of opened files +#LIBVIRTD_NOFILES_LIMIT=2048 diff --git a/daemon/libvirtd.upstart b/daemon/libvirtd.upstart index f51701a..e620f41 100644 --- a/daemon/libvirtd.upstart +++ b/daemon/libvirtd.upstart @@ -31,6 +31,11 @@ script ulimit -c "$DAEMON_COREFILE_LIMIT" fi
+ # LIBVIRTD_NOFILES_LIMIT from /etc/sysconfig/libvirtd is not handled + # automatically + if [ -n "$LIBVIRTD_NOFILES_LIMIT" ]; then + ulimit -n "$LIBVIRTD_NOFILES_LIMIT" + fi mkdir -p /var/cache/libvirt rm -rf /var/cache/libvirt/*
ACK Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (2)
-
Daniel P. Berrange
-
Michal Privoznik