Re: [libvirt] Ongoing work on lock contention in qemu driver?
On Wed, May 22, 2013 at 7:31 PM, Peter Feiner <peter(a)gridcentric.ca> wrote:
Since some security driver operations are costly, I think it's
worthwhile to reduce the scope of the security manager lock or
increase the granularity by introducing more locks. After a cursory
look, the security manager lock seems to have a much broader scope
than necessary. The system / library calls underlying the security
drivers are all thread safe (e.g., defining apparmor security profiles
or chowning disk files), so a global lock isn't strictly necessary.
Moreover, since most virSecurity calls are made whilst a virDomainObj
lock is held and the security calls are generally domain specific,
*most* of the security calls are probably thread safe in the absence
of the global security manager lock. Obviously some work will have to
be done to see where the security lock actually matters and some
finer-grained locks will have to be introduced to handle these
situations.
To verify that this is worthwhile, I disabled the apparmor driver
entirely. My 20 VM creation test ran about 10s faster (down from 35s
to 25s).
After giving this approach a little more thought, I think an
incremental series of patches is a good way to go. The responsibility
of locking could be pushed down into the security drivers. At first,
all of the drivers would lock where their managers' locked. Then each
driver could be updated to do more fine-grained locking. I'm going to
work on a patch to push the locking down into the drivers, then I'm
going to work on a patch for better locking in the apparmor driver.
I also think it's worthwhile to eliminate locking from the the
virDomainObjList lookups and traversals. Since virDomainObjLists are
accessed in a bunch of places, I think it's a good defensive idea to
decouple the performance of these accesses from virDomainObj locks,
which are held during potentially long-running operations like domain
creation. An easy way to divorce virDomainObjListSearchName from the
virDomainObj lock would be to keep a copy of the domain names in the
virDomainObjList and protect that list with the virDomainObjList lock.
After removing the security driver contention, this was still a
substantial bottleneck: virConnectDefineXML could still take a few
seconds. I removed the contention by keeping a copy of the domain
definition's name in the domain object. Since the name is immutable
and the domain object is protected by the list lock, the list
traversal can read the name without taking any additional locks. This
patch reduced virConnectDefineXML to tens of milliseconds.