Hi, http://bugs.debian.org/663931 is a bug I'm hitting, where virt-manager times out on the initial connection to libvirt.

The basic problem is that, while checking storage volumes, virt-manager causes libvirt to call "udevadm settle". There's an interaction where libvirt's earlier use of network namespaces (to probe LXC features) had caused some uevents to be sent that get filtered out before they reach udev. This confuses "udevadm settle" a bit, and so it sits there waiting for a 2-3 minute built-in timeout before returning. Eventually libvirtd prints: 2012-04-22 18:22:18.678+0000: 30503: warning : virKeepAliveTimer:182 : No response from client 0x7feec4003630 after 5 keepalive messages in 30 seconds and virt-manager prints: 2012-04-22 18:22:18.931+0000: 30647: warning : virKeepAliveSend:128 : Failed to send keepalive response to client 0x25004e0 and the connection gets dropped. One workaround could be to specify a shorter timeout when doing the settle. The patch appended below allows virt-manager to work, although the connection still has to wait for the 10 second timeout before it succeeds. I don't know what a better solution would be, though. It seems the udevadm behavior might not be considered a bug from the udev/kernel point of view: https://lkml.org/lkml/2012/4/22/60 I'm using Linux 3.2.14 with libvirt 0.9.11. You can trigger the udevadm issue using a program I posted at the Debian bug report link above. -jim >From 17e5b9ebab76acb0d711e8bc308023372fbc4180 Mon Sep 17 00:00:00 2001 From: Jim Paris <jim(a)jtan.com&gt; Date: Sun, 22 Apr 2012 14:35:47 -0400 Subject: [PATCH] shorten udevadmin settle timeout Otherwise, udevadmin settle can take so long that connections from e.g. virt-manager will get closed. --- src/util/util.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/util/util.c b/src/util/util.c index 6e041d6..dfe458e 100644 --- a/src/util/util.c +++ b/src/util/util.c @@ -2593,9 +2593,9 @@ virFileFindMountPoint(const char *type ATTRIBUTE_UNUSED) void virFileWaitForDevices(void) { # ifdef UDEVADM - const char *const settleprog[] = { UDEVADM, "settle", NULL }; + const char *const settleprog[] = { UDEVADM, "settle", "--timeout", "10", NULL }; # else - const char *const settleprog[] = { UDEVSETTLE, NULL }; + const char *const settleprog[] = { UDEVSETTLE, "--timeout", "10", NULL }; # endif int exitstatus; -- 1.7.7 -- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

Hi, On Sun, Apr 22, 2012 at 02:41:54PM -0400, Jim Paris wrote: > Hi, > > http://bugs.debian.org/663931 is a bug I'm hitting, where virt-manager > times out on the initial connection to libvirt. I reassigned the bug back to libvirt. I still wonder what triggers this though for some users but not for others? Cheers, -- Guido > > The basic problem is that, while checking storage volumes, > virt-manager causes libvirt to call "udevadm settle". There's an > interaction where libvirt's earlier use of network namespaces (to probe > LXC features) had caused some uevents to be sent that get filtered out > before they reach udev. This confuses "udevadm settle" a bit, and so > it sits there waiting for a 2-3 minute built-in timeout before returning. > Eventually libvirtd prints: > 2012-04-22 18:22:18.678+0000: 30503: warning : virKeepAliveTimer:182 : No response from client 0x7feec4003630 after 5 keepalive messages in 30 seconds > and virt-manager prints: > 2012-04-22 18:22:18.931+0000: 30647: warning : virKeepAliveSend:128 : Failed to send keepalive response to client 0x25004e0 > and the connection gets dropped. > > One workaround could be to specify a shorter timeout when doing the > settle. The patch appended below allows virt-manager to work, > although the connection still has to wait for the 10 second timeout > before it succeeds. I don't know what a better solution would be, > though. It seems the udevadm behavior might not be considered a bug > from the udev/kernel point of view: > https://lkml.org/lkml/2012/4/22/60 > > I'm using Linux 3.2.14 with libvirt 0.9.11. You can trigger the > udevadm issue using a program I posted at the Debian bug report link > above. > > -jim > > > From 17e5b9ebab76acb0d711e8bc308023372fbc4180 Mon Sep 17 00:00:00 2001 > From: Jim Paris&lt;jim(a)jtan.com&gt; > Date: Sun, 22 Apr 2012 14:35:47 -0400 > Subject: [PATCH] shorten udevadmin settle timeout > > Otherwise, udevadmin settle can take so long that connections from > e.g. virt-manager will get closed. > --- > src/util/util.c | 4 ++-- > 1 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/src/util/util.c b/src/util/util.c > index 6e041d6..dfe458e 100644 > --- a/src/util/util.c > +++ b/src/util/util.c > @@ -2593,9 +2593,9 @@ virFileFindMountPoint(const char *type ATTRIBUTE_UNUSED) > void virFileWaitForDevices(void) > { > # ifdef UDEVADM > - const char *const settleprog[] = { UDEVADM, "settle", NULL }; > + const char *const settleprog[] = { UDEVADM, "settle", "--timeout", "10", NULL };

Osier Yang wrote: > On 2012年04月24日 03:47, Guido Günther wrote: >> Hi, >> On Sun, Apr 22, 2012 at 02:41:54PM -0400, Jim Paris wrote: >>> Hi, >>> >>> http://bugs.debian.org/663931 is a bug I'm hitting, where virt-manager >>> times out on the initial connection to libvirt. >> >> I reassigned the bug back to libvirt. I still wonder what triggers this >> though for some users but not for others? >> Cheers, >> -- Guido >> >>> >>> The basic problem is that, while checking storage volumes, >>> virt-manager causes libvirt to call "udevadm settle". There's an >>> interaction where libvirt's earlier use of network namespaces (to probe >>> LXC features) had caused some uevents to be sent that get filtered out >>> before they reach udev. This confuses "udevadm settle" a bit, and so >>> it sits there waiting for a 2-3 minute built-in timeout before returning. >>> Eventually libvirtd prints: >>> 2012-04-22 18:22:18.678+0000: 30503: warning : virKeepAliveTimer:182 : No response from client 0x7feec4003630 after 5 keepalive messages in 30 seconds >>> and virt-manager prints: >>> 2012-04-22 18:22:18.931+0000: 30647: warning : virKeepAliveSend:128 : Failed to send keepalive response to client 0x25004e0 >>> and the connection gets dropped. >>> >>> One workaround could be to specify a shorter timeout when doing the >>> settle. The patch appended below allows virt-manager to work, >>> although the connection still has to wait for the 10 second timeout >>> before it succeeds. I don't know what a better solution would be, >>> though. It seems the udevadm behavior might not be considered a bug >> >from the udev/kernel point of view: >>> https://lkml.org/lkml/2012/4/22/60 >>> >>> I'm using Linux 3.2.14 with libvirt 0.9.11. You can trigger the >>> udevadm issue using a program I posted at the Debian bug report link >>> above. >>> >>> -jim >>> >>>> From 17e5b9ebab76acb0d711e8bc308023372fbc4180 Mon Sep 17 00:00:00 2001 >>> From: Jim Paris&lt;jim(a)jtan.com&gt; >>> Date: Sun, 22 Apr 2012 14:35:47 -0400 >>> Subject: [PATCH] shorten udevadmin settle timeout >>> >>> Otherwise, udevadmin settle can take so long that connections from >>> e.g. virt-manager will get closed. >>> --- >>> src/util/util.c | 4 ++-- >>> 1 files changed, 2 insertions(+), 2 deletions(-) >>> >>> diff --git a/src/util/util.c b/src/util/util.c >>> index 6e041d6..dfe458e 100644 >>> --- a/src/util/util.c >>> +++ b/src/util/util.c >>> @@ -2593,9 +2593,9 @@ virFileFindMountPoint(const char *type ATTRIBUTE_UNUSED) >>> void virFileWaitForDevices(void) >>> { >>> # ifdef UDEVADM >>> - const char *const settleprog[] = { UDEVADM, "settle", NULL }; >>> + const char *const settleprog[] = { UDEVADM, "settle", "--timeout", "10", NULL }; > > Though I don't have a good idea to fix it either, I guess this > change could cause "lvremove" to fail again for the udev race. > > See BZs: > > https://bugzilla.redhat.com/show_bug.cgi?id=702260 > https://bugzilla.redhat.com/show_bug.cgi?id=570359 It seems that those bugs were caused by something like 1. open(lv, O_RDWR) 2. close(lv) 3. system("lvremove ...") where udev would fire off a command between 2 and 3 that caused 3 to fail. Adding "udevadm settle" as step 2.5 is a good way to wait for that command to finish, but: - it doesn't necessarily fix the issue; something could easily re-open the device between 2.5 and 3 and cause the same failure.

- the race condition sounds like it was a short window, and sometimes the original sequence would still work even without the settle. That would suggest to me that a timeout of 10s is still plenty long. A few thoughts: - For lvremove: can we try a short timeout (3 seconds), then if the lvremove still fails, try again with the default udevadm timeout (120 seconds)? - Even in that case, we need to fix libvirtd to not kill the connection after 30 seconds when it's libvirtd's fault that the connection is blocked for so long anyway.

- When connecting with virt-manager, is the udevadm settle really necessary? We're not calling lvremove.