On Fri, Dec 26, 2014 at 03:54:12PM +0530, Prerna Saxena wrote:
I recently encountered a situation where an unclean ebtables
shutdown
caused /var/lib/ebtables/lock to be left behind. When libvirtd was
started on such a system, it caused libvirtd to "hang". Reason:
So you are saying that the lockfile exists, but no ebtables process
is still running ?
While probing to check if locking was supported, libvirt runs this
command synchronously :
# /usr/sbin/ebtables --concurrent -L
And this seemed to go on with msgs like :
Trying to obtain lock /var/lib/ebtables/lock
Trying to obtain lock /var/lib/ebtables/lock
Trying to obtain lock /var/lib/ebtables/lock
Result:
Libvirtd never recovered from this scenario, and the system was
essentially unable to start any VMs.
The following patch fixes this scenario:
-----------------------------------------------------------
>From ec245eccc03e8a69dc2c2e6edbf30a7b34eb74d0 Mon Sep 17 00:00:00 2001
From: Prerna Saxena <prerna(a)linux.vnet.ibm.com>
Date: Fri, 26 Dec 2014 15:24:45 -0500
Subject: [PATCH] Firewall : let libvirtd proceed after verifying valid locking
args.
Commit dc33e6e4a5a5d42 introduces locking args to be run with [eb/ip]tables to
determine whether locking is supported. However, this command needs to be
run asynchronously ( as against its present synchronous run), and needs to be
gracefully terminated once the job is done. Else it can potentially stall libvirtd
with messages like :
"Trying to acquire lock ..."
This doesn't really fully fix the problem - the stale lockfile still
exists, so as soon as you try to start a VM with filters, we're still
going to get stuck. You're just delaying the point at which anyone
will see the problem, which I'm not convinced is good as the admin
is far more likely to see the problem during libvirtd startup, than
when launching a VM.
Really, the fault lies with ebtables for just using the existance of
the lock file as its mutex. It should be fixed to use fcntl locks
on the file as its mutex, which guarantees you can never get any stale
lock files even when exiting uncleanly.
Regards,
Daniel
--
|:
http://berrange.com -o-
http://www.flickr.com/photos/dberrange/ :|
|:
http://libvirt.org -o-
http://virt-manager.org :|
|:
http://autobuild.org -o-
http://search.cpan.org/~danberr/ :|
|:
http://entangle-photo.org -o-
http://live.gnome.org/gtk-vnc :|