[libvirt] [PATCH 0/3] Fix scalability of LXC containers
From: "Daniel P. Berrange" <berrange@redhat.com> Since we switched to use systemd for creating cgroups we have a scalability problem with LXC out of the box. Only ~230 containers can be created before we hit the default dbus-daemon connection limit. This series makes libvirt_lxc close its dbus connection once the guest has started, avoiding the problem. Daniel P. Berrange (3): Allow use of a private dbus bus connection Add a method for closing the dbus system bus connection Make LXC controller use a private dbus connection & close it src/Makefile.am | 1 + src/libvirt_private.syms | 2 ++ src/lxc/lxc_controller.c | 9 +++++++++ src/util/virdbus.c | 28 +++++++++++++++++++++++++++- src/util/virdbus.h | 3 +++ 5 files changed, 42 insertions(+), 1 deletion(-) -- 1.8.3.1
From: "Daniel P. Berrange" <berrange@redhat.com> The dbus_bus_get() function returns a shared bus connection that all libraries in a process can use. You are forbidden from calling close on this connection though, since you can never know if any other code might be using it. Add an option to use private dbus bus connections, if the app wants to be able to close the connection. Signed-off-by: Daniel P. Berrange <berrange@redhat.com> --- src/libvirt_private.syms | 1 + src/util/virdbus.c | 16 +++++++++++++++- src/util/virdbus.h | 2 ++ 3 files changed, 18 insertions(+), 1 deletion(-) diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index 844ca29..b2b1fa8 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -1108,6 +1108,7 @@ virDBusHasSystemBus; virDBusMessageDecode; virDBusMessageEncode; virDBusMessageRead; +virDBusSetSharedBus; # util/virdnsmasq.h diff --git a/src/util/virdbus.c b/src/util/virdbus.c index 3bd339a..3c6007e 100644 --- a/src/util/virdbus.c +++ b/src/util/virdbus.c @@ -32,6 +32,7 @@ #ifdef WITH_DBUS +static bool sharedBus = true; static DBusConnection *systembus = NULL; static DBusConnection *sessionbus = NULL; static virOnceControl systemonce = VIR_ONCE_CONTROL_INITIALIZER; @@ -43,6 +44,11 @@ static dbus_bool_t virDBusAddWatch(DBusWatch *watch, void *data); static void virDBusRemoveWatch(DBusWatch *watch, void *data); static void virDBusToggleWatch(DBusWatch *watch, void *data); +void virDBusSetSharedBus(bool shared) +{ + sharedBus = shared; +} + static DBusConnection *virDBusBusInit(DBusBusType type, DBusError *dbuserr) { DBusConnection *bus; @@ -52,7 +58,10 @@ static DBusConnection *virDBusBusInit(DBusBusType type, DBusError *dbuserr) dbus_threads_init_default(); dbus_error_init(dbuserr); - if (!(bus = dbus_bus_get(type, dbuserr))) + bus = sharedBus ? + dbus_bus_get(type, dbuserr) : + dbus_bus_get_private(type, dbuserr); + if (!bus) return NULL; dbus_connection_set_exit_on_disconnect(bus, FALSE); @@ -1281,6 +1290,11 @@ int virDBusIsServiceEnabled(const char *name) #else /* ! WITH_DBUS */ +void virDBusSetSharedBus(bool shared ATTRIBUTE_UNUSED) +{ + /* nothing */ +} + DBusConnection *virDBusGetSystemBus(void) { virReportError(VIR_ERR_INTERNAL_ERROR, diff --git a/src/util/virdbus.h b/src/util/virdbus.h index 194a01a..125a405 100644 --- a/src/util/virdbus.h +++ b/src/util/virdbus.h @@ -31,6 +31,8 @@ # endif # include "internal.h" +void virDBusSetSharedBus(bool shared); + DBusConnection *virDBusGetSystemBus(void); bool virDBusHasSystemBus(void); DBusConnection *virDBusGetSessionBus(void); -- 1.8.3.1
On 10/11/2013 09:19 AM, Daniel P. Berrange wrote:
From: "Daniel P. Berrange" <berrange@redhat.com>
The dbus_bus_get() function returns a shared bus connection that all libraries in a process can use. You are forbidden from calling close on this connection though, since you can never know if any other code might be using it.
Doesn't that mean that multiple calls to dbus_bus_get() return the same shared bus when possible rather than creating a new one? Thus, how would you run out of resources if no new bus is created on the second call? Or is it a case of a new bus per set of different arguments, where we are consuming lots of shared buses by passing different arguments every time?
Add an option to use private dbus bus connections, if the app wants to be able to close the connection.
Signed-off-by: Daniel P. Berrange <berrange@redhat.com> --- src/libvirt_private.syms | 1 + src/util/virdbus.c | 16 +++++++++++++++- src/util/virdbus.h | 2 ++ 3 files changed, 18 insertions(+), 1 deletion(-)
At any rate, this patch makes sense. ACK. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Fri, Oct 11, 2013 at 10:36:37AM -0600, Eric Blake wrote:
On 10/11/2013 09:19 AM, Daniel P. Berrange wrote:
From: "Daniel P. Berrange" <berrange@redhat.com>
The dbus_bus_get() function returns a shared bus connection that all libraries in a process can use. You are forbidden from calling close on this connection though, since you can never know if any other code might be using it.
Doesn't that mean that multiple calls to dbus_bus_get() return the same shared bus when possible rather than creating a new one? Thus, how would you run out of resources if no new bus is created on the second call? Or is it a case of a new bus per set of different arguments, where we are consuming lots of shared buses by passing different arguments every time?
The issue is that the call is made in the libvirt_lxc process, and there's one of those per guest. So we have multiple connections. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 10/11/2013 10:50 AM, Daniel P. Berrange wrote:
On Fri, Oct 11, 2013 at 10:36:37AM -0600, Eric Blake wrote:
On 10/11/2013 09:19 AM, Daniel P. Berrange wrote:
From: "Daniel P. Berrange" <berrange@redhat.com>
The dbus_bus_get() function returns a shared bus connection that all libraries in a process can use. You are forbidden from calling close on this connection though, since you can never know if any other code might be using it.
Doesn't that mean that multiple calls to dbus_bus_get() return the same shared bus when possible rather than creating a new one? Thus, how would you run out of resources if no new bus is created on the second call? Or is it a case of a new bus per set of different arguments, where we are consuming lots of shared buses by passing different arguments every time?
The issue is that the call is made in the libvirt_lxc process, and there's one of those per guest. So we have multiple connections.
Ah, so the _parent_ process (libvirtd) continues to share the bus, but now has multiple long-lived lxc_process children. While we can't close the bus in normal libvirt.so usage (because we don't know what else is linked in to the app using the shared library), we CAN do it in lxc_process (where we know the entire app, and know that lxc_process doesn't care about dbus after setup is complete). Makes more sense. My ACKs on code quality still stand, and now you can add my ACK to the design :) -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
From: "Daniel P. Berrange" <berrange@redhat.com> If the dbus system bus connection is marked as private, then allow it to be closed. Signed-off-by: Daniel P. Berrange <berrange@redhat.com> --- src/libvirt_private.syms | 1 + src/util/virdbus.c | 12 ++++++++++++ src/util/virdbus.h | 1 + 3 files changed, 14 insertions(+) diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index b2b1fa8..7ee538c 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -1102,6 +1102,7 @@ virConfWriteMem; # util/virdbus.h virDBusCallMethod; +virDBusCloseSystemBus; virDBusGetSessionBus; virDBusGetSystemBus; virDBusHasSystemBus; diff --git a/src/util/virdbus.c b/src/util/virdbus.c index 3c6007e..dc4ace0 100644 --- a/src/util/virdbus.c +++ b/src/util/virdbus.c @@ -122,6 +122,14 @@ virDBusHasSystemBus(void) } +void virDBusCloseSystemBus(void) +{ + if (systembus && !sharedBus) { + dbus_connection_close(systembus); + systembus = NULL; + } +} + static void virDBusSessionBusInit(void) { sessionbus = virDBusBusInit(DBUS_BUS_SESSION, &sessiondbuserr); @@ -1310,6 +1318,10 @@ virDBusHasSystemBus(void) return false; } +void virDBusCloseSystemBus(void) +{ + /* nothing */ +} DBusConnection *virDBusGetSessionBus(void) { diff --git a/src/util/virdbus.h b/src/util/virdbus.h index 125a405..979c566 100644 --- a/src/util/virdbus.h +++ b/src/util/virdbus.h @@ -35,6 +35,7 @@ void virDBusSetSharedBus(bool shared); DBusConnection *virDBusGetSystemBus(void); bool virDBusHasSystemBus(void); +void virDBusCloseSystemBus(void); DBusConnection *virDBusGetSessionBus(void); int virDBusCallMethod(DBusConnection *conn, -- 1.8.3.1
On 10/11/2013 09:19 AM, Daniel P. Berrange wrote:
From: "Daniel P. Berrange" <berrange@redhat.com>
If the dbus system bus connection is marked as private, then allow it to be closed.
Signed-off-by: Daniel P. Berrange <berrange@redhat.com> --- src/libvirt_private.syms | 1 + src/util/virdbus.c | 12 ++++++++++++ src/util/virdbus.h | 1 + 3 files changed, 14 insertions(+)
ACK. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
From: "Daniel P. Berrange" <berrange@redhat.com> The LXC controller uses dbus to talk to systemd to create cgroups. This means that each LXC controller instance has a dbus connection. The DBus daemon is limited to 256 connections by default and we want to be able to run many 1000 of containers. While the dbus limit could be raised in the config files, it is simpler to make libvirt LXC controller close its dbus connection once everything is configured. Signed-off-by: Daniel P. Berrange <berrange@redhat.com> --- src/Makefile.am | 1 + src/lxc/lxc_controller.c | 9 +++++++++ 2 files changed, 10 insertions(+) diff --git a/src/Makefile.am b/src/Makefile.am index 201c268..e58d408 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -2381,6 +2381,7 @@ libvirt_lxc_CFLAGS = \ $(PIE_CFLAGS) \ $(LIBNL_CFLAGS) \ $(FUSE_CFLAGS) \ + $(DBUS_CFLAGS) \ $(NULL) if WITH_BLKID libvirt_lxc_CFLAGS += $(BLKID_CFLAGS) diff --git a/src/lxc/lxc_controller.c b/src/lxc/lxc_controller.c index bb33c28..b881f17 100644 --- a/src/lxc/lxc_controller.c +++ b/src/lxc/lxc_controller.c @@ -64,6 +64,7 @@ #include "virrandom.h" #include "virprocess.h" #include "virnuma.h" +#include "virdbus.h" #include "rpc/virnetserver.h" #include "virstring.h" @@ -2200,6 +2201,12 @@ virLXCControllerRun(virLXCControllerPtr ctrl) if (virLXCControllerConsoleSetNonblocking(&(ctrl->consoles[i])) < 0) goto cleanup; + /* We must not hold open a dbus connection for life + * of LXC instance, since dbus-daemon is limited to + * only a few 100 connections by default + */ + virDBusCloseSystemBus(); + rc = virLXCControllerMain(ctrl); virLXCControllerEventSendExit(ctrl, rc); @@ -2351,6 +2358,8 @@ int main(int argc, char *argv[]) virEventRegisterDefaultImpl(); + virDBusSetSharedBus(false); + if (!(ctrl = virLXCControllerNew(name))) goto cleanup; -- 1.8.3.1
On 10/11/2013 09:19 AM, Daniel P. Berrange wrote:
From: "Daniel P. Berrange" <berrange@redhat.com>
The LXC controller uses dbus to talk to systemd to create cgroups. This means that each LXC controller instance has a dbus connection. The DBus daemon is limited to 256 connections by default and we want to be able to run many 1000 of containers.
While the dbus limit could be raised in the config files, it is simpler to make libvirt LXC controller close its dbus connection once everything is configured.
Signed-off-by: Daniel P. Berrange <berrange@redhat.com> --- src/Makefile.am | 1 + src/lxc/lxc_controller.c | 9 +++++++++ 2 files changed, 10 insertions(+)
ACK. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
participants (2)
-
Daniel P. Berrange -
Eric Blake