10:19 a.m.
From: "Daniel P. Berrange" <berrange(a)redhat.com>
Since we switched to use systemd for creating cgroups we have a
scalability problem with LXC out of the box. Only ~230 containers
can be created before we hit the default dbus-daemon connection
limit. This series makes libvirt_lxc close its dbus connection
once the guest has started, avoiding the problem.
Daniel P. Berrange (3):
Allow use of a private dbus bus connection
Add a method for closing the dbus system bus connection
Make LXC controller use a private dbus connection & close it
src/Makefile.am | 1 +
src/libvirt_private.syms | 2 ++
src/lxc/lxc_controller.c | 9 +++++++++
src/util/virdbus.c | 28 +++++++++++++++++++++++++++-
src/util/virdbus.h | 3 +++
5 files changed, 42 insertions(+), 1 deletion(-)
--
1.8.3.1
10:19 a.m.
New subject: [libvirt] [PATCH 1/3] Allow use of a private dbus bus connection
From: "Daniel P. Berrange" <berrange(a)redhat.com>
The dbus_bus_get() function returns a shared bus connection that
all libraries in a process can use. You are forbidden from calling
close on this connection though, since you can never know if any
other code might be using it.
Add an option to use private dbus bus connections, if the app
wants to be able to close the connection.
Signed-off-by: Daniel P. Berrange <berrange(a)redhat.com>
---
src/libvirt_private.syms | 1 +
src/util/virdbus.c | 16 +++++++++++++++-
src/util/virdbus.h | 2 ++
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms
index 844ca29..b2b1fa8 100644
--- a/src/libvirt_private.syms
+++ b/src/libvirt_private.syms
@@ -1108,6 +1108,7 @@ virDBusHasSystemBus;
virDBusMessageDecode;
virDBusMessageEncode;
virDBusMessageRead;
+virDBusSetSharedBus;
# util/virdnsmasq.h
diff --git a/src/util/virdbus.c b/src/util/virdbus.c
index 3bd339a..3c6007e 100644
--- a/src/util/virdbus.c
+++ b/src/util/virdbus.c
@@ -32,6 +32,7 @@
#ifdef WITH_DBUS
+static bool sharedBus = true;
static DBusConnection *systembus = NULL;
static DBusConnection *sessionbus = NULL;
static virOnceControl systemonce = VIR_ONCE_CONTROL_INITIALIZER;
@@ -43,6 +44,11 @@ static dbus_bool_t virDBusAddWatch(DBusWatch *watch, void *data);
static void virDBusRemoveWatch(DBusWatch *watch, void *data);
static void virDBusToggleWatch(DBusWatch *watch, void *data);
+void virDBusSetSharedBus(bool shared)
+{
+ sharedBus = shared;
+}
+
static DBusConnection *virDBusBusInit(DBusBusType type, DBusError *dbuserr)
{
DBusConnection *bus;
@@ -52,7 +58,10 @@ static DBusConnection *virDBusBusInit(DBusBusType type, DBusError
*dbuserr)
dbus_threads_init_default();
dbus_error_init(dbuserr);
- if (!(bus = dbus_bus_get(type, dbuserr)))
+ bus = sharedBus ?
+ dbus_bus_get(type, dbuserr) :
+ dbus_bus_get_private(type, dbuserr);
+ if (!bus)
return NULL;
dbus_connection_set_exit_on_disconnect(bus, FALSE);
@@ -1281,6 +1290,11 @@ int virDBusIsServiceEnabled(const char *name)
#else /* ! WITH_DBUS */
+void virDBusSetSharedBus(bool shared ATTRIBUTE_UNUSED)
+{
+ /* nothing */
+}
+
DBusConnection *virDBusGetSystemBus(void)
{
virReportError(VIR_ERR_INTERNAL_ERROR,
diff --git a/src/util/virdbus.h b/src/util/virdbus.h
index 194a01a..125a405 100644
--- a/src/util/virdbus.h
+++ b/src/util/virdbus.h
@@ -31,6 +31,8 @@
# endif
# include "internal.h"
+void virDBusSetSharedBus(bool shared);
+
DBusConnection *virDBusGetSystemBus(void);
bool virDBusHasSystemBus(void);
DBusConnection *virDBusGetSessionBus(void);
--
1.8.3.1
11:36 a.m.
New subject: [libvirt] [PATCH 1/3] Allow use of a private dbus bus connection
On 10/11/2013 09:19 AM, Daniel P. Berrange wrote:
From: "Daniel P. Berrange" <berrange(a)redhat.com>
The dbus_bus_get() function returns a shared bus connection that
all libraries in a process can use. You are forbidden from calling
close on this connection though, since you can never know if any
other code might be using it.
Doesn't that mean that multiple calls to dbus_bus_get() return the same
shared bus when possible rather than creating a new one? Thus, how
would you run out of resources if no new bus is created on the second
call? Or is it a case of a new bus per set of different arguments,
where we are consuming lots of shared buses by passing different
arguments every time?
Add an option to use private dbus bus connections, if the app
wants to be able to close the connection.
Signed-off-by: Daniel P. Berrange <berrange(a)redhat.com>
---
src/libvirt_private.syms | 1 +
src/util/virdbus.c | 16 +++++++++++++++-
src/util/virdbus.h | 2 ++
3 files changed, 18 insertions(+), 1 deletion(-)
At any rate, this patch makes sense. ACK.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
11:50 a.m.
New subject: [libvirt] [PATCH 1/3] Allow use of a private dbus bus connection
On Fri, Oct 11, 2013 at 10:36:37AM -0600, Eric Blake wrote:
On 10/11/2013 09:19 AM, Daniel P. Berrange wrote:
> From: "Daniel P. Berrange" <berrange(a)redhat.com>
>
> The dbus_bus_get() function returns a shared bus connection that
> all libraries in a process can use. You are forbidden from calling
> close on this connection though, since you can never know if any
> other code might be using it.
Doesn't that mean that multiple calls to dbus_bus_get() return the same
shared bus when possible rather than creating a new one? Thus, how
would you run out of resources if no new bus is created on the second
call? Or is it a case of a new bus per set of different arguments,
where we are consuming lots of shared buses by passing different
arguments every time?
The issue is that the call is made in the libvirt_lxc process,
and there's one of those per guest. So we have multiple connections.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
11:53 a.m.
New subject: [libvirt] [PATCH 1/3] Allow use of a private dbus bus connection
On 10/11/2013 10:50 AM, Daniel P. Berrange wrote:
On Fri, Oct 11, 2013 at 10:36:37AM -0600, Eric Blake wrote:
> On 10/11/2013 09:19 AM, Daniel P. Berrange wrote:
>> From: "Daniel P. Berrange" <berrange(a)redhat.com>
>>
>> The dbus_bus_get() function returns a shared bus connection that
>> all libraries in a process can use. You are forbidden from calling
>> close on this connection though, since you can never know if any
>> other code might be using it.
>
> Doesn't that mean that multiple calls to dbus_bus_get() return the same
> shared bus when possible rather than creating a new one? Thus, how
> would you run out of resources if no new bus is created on the second
> call? Or is it a case of a new bus per set of different arguments,
> where we are consuming lots of shared buses by passing different
> arguments every time?
The issue is that the call is made in the libvirt_lxc process,
and there's one of those per guest. So we have multiple connections.
Ah, so the _parent_ process (libvirtd) continues to share the bus, but
now has multiple long-lived lxc_process children. While we can't close
the bus in normal libvirt.so usage (because we don't know what else is
linked in to the app using the shared library), we CAN do it in
lxc_process (where we know the entire app, and know that lxc_process
doesn't care about dbus after setup is complete). Makes more sense. My
ACKs on code quality still stand, and now you can add my ACK to the
design :)
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
10:19 a.m.
New subject: [libvirt] [PATCH 2/3] Add a method for closing the dbus system bus connection
From: "Daniel P. Berrange" <berrange(a)redhat.com>
If the dbus system bus connection is marked as private, then
allow it to be closed.
Signed-off-by: Daniel P. Berrange <berrange(a)redhat.com>
---
src/libvirt_private.syms | 1 +
src/util/virdbus.c | 12 ++++++++++++
src/util/virdbus.h | 1 +
3 files changed, 14 insertions(+)
diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms
index b2b1fa8..7ee538c 100644
--- a/src/libvirt_private.syms
+++ b/src/libvirt_private.syms
@@ -1102,6 +1102,7 @@ virConfWriteMem;
# util/virdbus.h
virDBusCallMethod;
+virDBusCloseSystemBus;
virDBusGetSessionBus;
virDBusGetSystemBus;
virDBusHasSystemBus;
diff --git a/src/util/virdbus.c b/src/util/virdbus.c
index 3c6007e..dc4ace0 100644
--- a/src/util/virdbus.c
+++ b/src/util/virdbus.c
@@ -122,6 +122,14 @@ virDBusHasSystemBus(void)
}
+void virDBusCloseSystemBus(void)
+{
+ if (systembus && !sharedBus) {
+ dbus_connection_close(systembus);
+ systembus = NULL;
+ }
+}
+
static void virDBusSessionBusInit(void)
{
sessionbus = virDBusBusInit(DBUS_BUS_SESSION, &sessiondbuserr);
@@ -1310,6 +1318,10 @@ virDBusHasSystemBus(void)
return false;
}
+void virDBusCloseSystemBus(void)
+{
+ /* nothing */
+}
DBusConnection *virDBusGetSessionBus(void)
{
diff --git a/src/util/virdbus.h b/src/util/virdbus.h
index 125a405..979c566 100644
--- a/src/util/virdbus.h
+++ b/src/util/virdbus.h
@@ -35,6 +35,7 @@ void virDBusSetSharedBus(bool shared);
DBusConnection *virDBusGetSystemBus(void);
bool virDBusHasSystemBus(void);
+void virDBusCloseSystemBus(void);
DBusConnection *virDBusGetSessionBus(void);
int virDBusCallMethod(DBusConnection *conn,
--
1.8.3.1
11:39 a.m.
New subject: [libvirt] [PATCH 2/3] Add a method for closing the dbus system bus connection
On 10/11/2013 09:19 AM, Daniel P. Berrange wrote:
From: "Daniel P. Berrange" <berrange(a)redhat.com>
If the dbus system bus connection is marked as private, then
allow it to be closed.
Signed-off-by: Daniel P. Berrange <berrange(a)redhat.com>
---
src/libvirt_private.syms | 1 +
src/util/virdbus.c | 12 ++++++++++++
src/util/virdbus.h | 1 +
3 files changed, 14 insertions(+)
ACK.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
10:19 a.m.
New subject: [libvirt] [PATCH 3/3] Make LXC controller use a private dbus connection & close it
From: "Daniel P. Berrange" <berrange(a)redhat.com>
The LXC controller uses dbus to talk to systemd to create
cgroups. This means that each LXC controller instance has
a dbus connection. The DBus daemon is limited to 256
connections by default and we want to be able to run many
1000 of containers.
While the dbus limit could be raised in the config files,
it is simpler to make libvirt LXC controller close its
dbus connection once everything is configured.
Signed-off-by: Daniel P. Berrange <berrange(a)redhat.com>
---
src/Makefile.am | 1 +
src/lxc/lxc_controller.c | 9 +++++++++
2 files changed, 10 insertions(+)
diff --git a/src/Makefile.am b/src/Makefile.am
index 201c268..e58d408 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -2381,6 +2381,7 @@ libvirt_lxc_CFLAGS = \
$(PIE_CFLAGS) \
$(LIBNL_CFLAGS) \
$(FUSE_CFLAGS) \
+ $(DBUS_CFLAGS) \
$(NULL)
if WITH_BLKID
libvirt_lxc_CFLAGS += $(BLKID_CFLAGS)
diff --git a/src/lxc/lxc_controller.c b/src/lxc/lxc_controller.c
index bb33c28..b881f17 100644
--- a/src/lxc/lxc_controller.c
+++ b/src/lxc/lxc_controller.c
@@ -64,6 +64,7 @@
#include "virrandom.h"
#include "virprocess.h"
#include "virnuma.h"
+#include "virdbus.h"
#include "rpc/virnetserver.h"
#include "virstring.h"
@@ -2200,6 +2201,12 @@ virLXCControllerRun(virLXCControllerPtr ctrl)
if (virLXCControllerConsoleSetNonblocking(&(ctrl->consoles[i])) < 0)
goto cleanup;
+ /* We must not hold open a dbus connection for life
+ * of LXC instance, since dbus-daemon is limited to
+ * only a few 100 connections by default
+ */
+ virDBusCloseSystemBus();
+
rc = virLXCControllerMain(ctrl);
virLXCControllerEventSendExit(ctrl, rc);
@@ -2351,6 +2358,8 @@ int main(int argc, char *argv[])
virEventRegisterDefaultImpl();
+ virDBusSetSharedBus(false);
+
if (!(ctrl = virLXCControllerNew(name)))
goto cleanup;
--
1.8.3.1
11:45 a.m.
New subject: [libvirt] [PATCH 3/3] Make LXC controller use a private dbus connection & close it
On 10/11/2013 09:19 AM, Daniel P. Berrange wrote:
From: "Daniel P. Berrange" <berrange(a)redhat.com>
The LXC controller uses dbus to talk to systemd to create
cgroups. This means that each LXC controller instance has
a dbus connection. The DBus daemon is limited to 256
connections by default and we want to be able to run many
1000 of containers.
While the dbus limit could be raised in the config files,
it is simpler to make libvirt LXC controller close its
dbus connection once everything is configured.
Signed-off-by: Daniel P. Berrange <berrange(a)redhat.com>
---
src/Makefile.am | 1 +
src/lxc/lxc_controller.c | 9 +++++++++
2 files changed, 10 insertions(+)
ACK.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
4043
days inactive
4043
days old
8 comments
2 participants
participants (2)
-
Daniel P. Berrange -
Eric Blake