[libvirt] [PATCH 0/6] virObject for reference-counting

This series adds a virObject structure that manages reference-counting. Some notes about referece-counting introduced by this series: A thread owns a virObject by incrementing its reference-count by 1. If a thread owns a virObject, the virObject is guarenteed not be freed until the thread releases ownership by decrementing its reference-count by 1. A thread can't access a virObject after it releases the ownership of virObject because it can be freed at anytime. A thread can own a virObject legally in these ways: - a thread owns a virObject that it creates. - a thread owns a virObject if another thread passes the ownership to it. Example: qemuMonitorOpen - a thread gets a virObject from a container. Example: virDomainFindByUUID - a container owns a virObject by incrementing its reference-count by 1 before adding it to the container - if a virObject is removed from a container its reference-count must be decremented by 1 By following these rules, there is no need to protect operations on an object's reference-count by an external lock. (like in old ways virDomainObj lock protects qemu monitor's ref-count.) This series is a draft and posted for early review. Any comments are welcome. Hu Tao (6): Add virObject and virAtomic. use virObject to manage reference-count of qemud_client use virObject to manage reference-count of virDomainObj qemu: use virObject to manages reference-counting for qemu monitor remove qemuDomainObjEnterMonitorWithDriver and qemuDomainObjExitMonitorWithDriver remove qemuDomainObjBeginJobWithDriver daemon/dispatch.c | 2 - daemon/libvirtd.c | 62 ++++++-- daemon/libvirtd.h | 5 +- src/Makefile.am | 2 + src/conf/domain_conf.c | 56 ++++---- src/conf/domain_conf.h | 6 +- src/libvirt_private.syms | 5 + src/openvz/openvz_conf.c | 8 +- src/qemu/qemu_domain.c | 121 +-------------- src/qemu/qemu_domain.h | 8 +- src/qemu/qemu_driver.c | 358 ++++++++++++++++++++------------------------- src/qemu/qemu_hotplug.c | 90 ++++++------ src/qemu/qemu_migration.c | 101 ++++++------- src/qemu/qemu_monitor.c | 109 ++++++++------- src/qemu/qemu_monitor.h | 4 +- src/qemu/qemu_process.c | 99 ++++++------- src/util/viratomic.c | 46 ++++++ src/util/viratomic.h | 30 ++++ src/util/virobject.c | 52 +++++++ src/util/virobject.h | 39 +++++ src/vmware/vmware_conf.c | 2 +- 21 files changed, 628 insertions(+), 577 deletions(-) create mode 100644 src/util/viratomic.c create mode 100644 src/util/viratomic.h create mode 100644 src/util/virobject.c create mode 100644 src/util/virobject.h -- 1.7.3.1

virObject is the base struct that manages reference-counting for all structs that need the ability of reference-counting. virAtomic provides atomic operations which are thread-safe. --- src/Makefile.am | 2 + src/libvirt_private.syms | 5 ++++ src/util/viratomic.c | 46 ++++++++++++++++++++++++++++++++++++++++ src/util/viratomic.h | 30 ++++++++++++++++++++++++++ src/util/virobject.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++ src/util/virobject.h | 39 ++++++++++++++++++++++++++++++++++ 6 files changed, 174 insertions(+), 0 deletions(-) create mode 100644 src/util/viratomic.c create mode 100644 src/util/viratomic.h create mode 100644 src/util/virobject.c create mode 100644 src/util/virobject.h diff --git a/src/Makefile.am b/src/Makefile.am index 9b54679..9e74060 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -81,6 +81,8 @@ UTIL_SOURCES = \ util/util.c util/util.h \ util/xml.c util/xml.h \ util/virtaudit.c util/virtaudit.h \ + util/viratomic.c util/viratomic.h \ + util/virobject.c util/virobject.h \ util/virterror.c util/virterror_internal.h EXTRA_DIST += util/threads-pthread.c util/threads-win32.c diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index 65a86d3..98f8d6f 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -993,3 +993,8 @@ virXPathUInt; virXPathULong; virXPathULongHex; virXPathULongLong; + +# object.h +virObjectInit; +virObjectRef; +virObjectUnref; diff --git a/src/util/viratomic.c b/src/util/viratomic.c new file mode 100644 index 0000000..629f189 --- /dev/null +++ b/src/util/viratomic.c @@ -0,0 +1,46 @@ +/* + * viratomic.c: atomic operations + * + * Copyright (C) 2011 Hu Tao + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Authors: + * Hu Tao <hutao@cn.fujitsu.com> + */ + +#include "viratomic.h" + +#ifdef WIN32 +long virAtomicInc(long *value) +{ + return InterlockedIncrement(value); +} + +long virAtomicDec(long *value) +{ + return InterlockedDecrement(value); +} +#else /* WIN32 */ +long virAtomicInc(long *value) +{ + return __sync_add_and_fetch(value, 1); +} + +long virAtomicDec(long *value) +{ + return __sync_sub_and_fetch(value, 1); +} +#endif /* WIN32 */ diff --git a/src/util/viratomic.h b/src/util/viratomic.h new file mode 100644 index 0000000..a10cb65 --- /dev/null +++ b/src/util/viratomic.h @@ -0,0 +1,30 @@ +/* + * viratomic.h: atomic operations + * + * Copyright (C) 2011 Hu Tao + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Authors: + * Hu Tao <hutao@cn.fujitsu.com> + */ + +#ifndef __VIR_ATOMIC_H +#define __VIR_ATOMIC_H + +long virAtomicInc(long *value); +long virAtomicDec(long *value); + +#endif /* __VIR_ATOMIC_H */ diff --git a/src/util/virobject.c b/src/util/virobject.c new file mode 100644 index 0000000..aeedbef --- /dev/null +++ b/src/util/virobject.c @@ -0,0 +1,52 @@ +/* + * virobject.c: base object that manages reference-counting + * + * Copyright (C) 2011 Hu Tao + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Authors: + * Hu Tao <hutao@cn.fujitsu.com> + */ + +#include "viratomic.h" +#include "virobject.h" +#include "logging.h" + +int virObjectInit(virObjectPtr obj, void (*free)(virObjectPtr obj)) +{ + if (!free) { + VIR_ERROR0("method free is required."); + return -1; + } + + obj->ref = 1; + obj->free = free; + + return 0; +} + +void virObjectRef(virObjectPtr obj) +{ + sa_assert(obj->ref > 0); + virAtomicInc(&obj->ref); +} + +void virObjectUnref(virObjectPtr obj) +{ + sa_assert(obj->ref > 0); + if (virAtomicDec(&obj->ref) == 0) + obj->free(obj); +} diff --git a/src/util/virobject.h b/src/util/virobject.h new file mode 100644 index 0000000..cd7d3e8 --- /dev/null +++ b/src/util/virobject.h @@ -0,0 +1,39 @@ +/* + * virobject.h: base object that manages reference-counting + * + * Copyright (C) 2011 Hu Tao + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Authors: + * Hu Tao <hutao@cn.fujitsu.com> + */ + +#ifndef __VIR_OBJECT_H +#define __VIR_OBJECT_H + +typedef struct _virObject virObject; +typedef virObject *virObjectPtr; + +struct _virObject { + long ref; + void (*free)(virObjectPtr obj); +}; + +int virObjectInit(virObjectPtr obj, void (*free)(virObjectPtr obj)); +void virObjectRef(virObjectPtr obj); +void virObjectUnref(virObjectPtr obj); + +#endif /* __VIR_OBJECT_H */ -- 1.7.3.1 -- Thanks, Hu Tao

On 04/06/2011 01:19 AM, Hu Tao wrote:
virObject is the base struct that manages reference-counting for all structs that need the ability of reference-counting.
virAtomic provides atomic operations which are thread-safe. --- src/Makefile.am | 2 + src/libvirt_private.syms | 5 ++++ src/util/viratomic.c | 46 ++++++++++++++++++++++++++++++++++++++++ src/util/viratomic.h | 30 ++++++++++++++++++++++++++ src/util/virobject.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++ src/util/virobject.h | 39 ++++++++++++++++++++++++++++++++++ 6 files changed, 174 insertions(+), 0 deletions(-) create mode 100644 src/util/viratomic.c create mode 100644 src/util/viratomic.h create mode 100644 src/util/virobject.c create mode 100644 src/util/virobject.h
+++ b/src/libvirt_private.syms @@ -993,3 +993,8 @@ virXPathUInt; virXPathULong; virXPathULongHex; virXPathULongLong; + +# object.h
virobject.h, and float this section to appear before virtaudit.h (hmm, maybe we should do a preliminary patch to rename that to viraudit.h, so that we aren't mixing vir vs. virt in quite so many places).
+virObjectInit; +virObjectRef; +virObjectUnref;
Missing exports from viratomic.h.
diff --git a/src/util/viratomic.c b/src/util/viratomic.c new file mode 100644 index 0000000..629f189 --- /dev/null +++ b/src/util/viratomic.c @@ -0,0 +1,46 @@
+ +#ifdef WIN32 +long virAtomicInc(long *value) +{ + return InterlockedIncrement(value); +} + +long virAtomicDec(long *value) +{ + return InterlockedDecrement(value);
This is an OS-specific replacement.
+} +#else /* WIN32 */ +long virAtomicInc(long *value) +{ + return __sync_add_and_fetch(value, 1);
This is a gcc builtin, and will fail to compile with other C99 compilers. Meanwhile, won't the gcc builtin just work for mingw (that is, no need to use the OS-specific InterlockedIncrement if you have the compiler builtin instead). I think this file needs three implementations: #if defined __GNUC__ || <detect Intel's compiler, which also has these> use compiler builtins of __sync_add_and_fetch #elif defined WIN32 use OS primitives, like InterlockedIncrement #else we're hosed when it comes to lightweight versions, but we can still implement a heavyweight replacement that uses virMutex #endif
+++ b/src/util/viratomic.h @@ -0,0 +1,30 @@
+#ifndef __VIR_ATOMIC_H +#define __VIR_ATOMIC_H + +long virAtomicInc(long *value); +long virAtomicDec(long *value);
Mark both of these ATTRIBUTE_NONNULL(1) I'm debating whether they should also be marked ATTRIBUTE_RETURN_CHECK
+++ b/src/util/virobject.c @@ -0,0 +1,52 @@ + +#include "viratomic.h" +#include "virobject.h" +#include "logging.h" + +int virObjectInit(virObjectPtr obj, void (*free)(virObjectPtr obj))
You should declare a typedef: typedef void (*virObjectFreer)(virObjectPtr); then it becomes simpler to read: int virObjectInit(virObjectPtr obj, virObjectFreer f) Use a different name (I used freer/f above) to avoid -Wshadow warnings with free().
+{ + if (!free) {
Especially since shadowing means it's impossible to tell if this statement is always true (the address of free() exists) or conditional (there is a local variable named free shadowing the global function), and context-sensitive code reviews are tougher :)
+ VIR_ERROR0("method free is required."); + return -1; + }
Should this function also check and return -1 if obj->free was not NULL on entry (that is, guarantee that you can't initialize an object twice)?
+ + obj->ref = 1; + obj->free = free; + + return 0; +} + +void virObjectRef(virObjectPtr obj) +{ + sa_assert(obj->ref > 0);
This is useless. It only helps static analyzers (like clang),
+ virAtomicInc(&obj->ref);
but there's nothing to analyze that depends on knowing the value was positive. I'm debating whether to do checking. Maybe we should do: if (virAtomicInc(&obj->ref) < 2) VIR_WARN("invalid call to virObjectRef");
+void virObjectUnref(virObjectPtr obj) +{ + sa_assert(obj->ref > 0);
Again, I'm not sure that this buys anything. But we may want to do: int ref = virAtomicDec(&obj->ref); if (ref < 0) VIR_WARN("invalid call to virObjectUnref"); else if (ref == 0) obj->free(obj)
+ if (virAtomicDec(&obj->ref) == 0) + obj->free(obj); +} diff --git a/src/util/virobject.h b/src/util/virobject.h new file mode 100644 index 0000000..cd7d3e8 --- /dev/null +++ b/src/util/virobject.h
+ +typedef struct _virObject virObject; +typedef virObject *virObjectPtr; + +struct _virObject { + long ref; + void (*free)(virObjectPtr obj);
Is virObjectPtr the right thing to use here? If we assume that all clients will always have a virObjectPtr as their first member, then they can cast that back into the larger object. Is void* opaque any better, although that then it requires a third parameter for virObjectInit? Maybe it's worth some documentation on intended use in this header (am I getting this usage right? I haven't looked at how you used it later in the series, but am just guessing): struct _virFoo { virObject obj; /* Must be first member */ ... }; static void virFooFree(virObjectPtr obj) { virFooPtr foo = obj; ... } virFooPtr virFooNew(void) { virFooPtr foo; if (VIR_ALLOC(foo) < 0) { virReportOOMError(); return NULL; } virObjectInit(&foo->obj, virFooFree); ... return foo; }
+}; + +int virObjectInit(virObjectPtr obj, void (*free)(virObjectPtr obj));
ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) ATTRIBUTE_RETURN_CHECK
+void virObjectRef(virObjectPtr obj);
ATTRIBUTE_NONNULL(1) Also, should this return the current reference count? Not all callers will need it, but returning void is harsh if someone might be able to use it.
+void virObjectUnref(virObjectPtr obj);
Likewise. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

On Wed, Apr 06, 2011 at 02:19:57PM -0600, Eric Blake wrote:
On 04/06/2011 01:19 AM, Hu Tao wrote:
virObject is the base struct that manages reference-counting for all structs that need the ability of reference-counting.
virAtomic provides atomic operations which are thread-safe. --- src/Makefile.am | 2 + src/libvirt_private.syms | 5 ++++ src/util/viratomic.c | 46 ++++++++++++++++++++++++++++++++++++++++ src/util/viratomic.h | 30 ++++++++++++++++++++++++++ src/util/virobject.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++ src/util/virobject.h | 39 ++++++++++++++++++++++++++++++++++ 6 files changed, 174 insertions(+), 0 deletions(-) create mode 100644 src/util/viratomic.c create mode 100644 src/util/viratomic.h create mode 100644 src/util/virobject.c create mode 100644 src/util/virobject.h
+++ b/src/libvirt_private.syms @@ -993,3 +993,8 @@ virXPathUInt; virXPathULong; virXPathULongHex; virXPathULongLong; + +# object.h
virobject.h, and float this section to appear before virtaudit.h (hmm,
Oops, there are always things escape from under your nose:(
maybe we should do a preliminary patch to rename that to viraudit.h, so that we aren't mixing vir vs. virt in quite so many places).
+virObjectInit; +virObjectRef; +virObjectUnref;
Missing exports from viratomic.h.
Why the linker doesn't complain about this this time?
diff --git a/src/util/viratomic.c b/src/util/viratomic.c new file mode 100644 index 0000000..629f189 --- /dev/null +++ b/src/util/viratomic.c @@ -0,0 +1,46 @@
+ +#ifdef WIN32 +long virAtomicInc(long *value) +{ + return InterlockedIncrement(value); +} + +long virAtomicDec(long *value) +{ + return InterlockedDecrement(value);
This is an OS-specific replacement.
+} +#else /* WIN32 */ +long virAtomicInc(long *value) +{ + return __sync_add_and_fetch(value, 1);
This is a gcc builtin, and will fail to compile with other C99
Yes.
compilers. Meanwhile, won't the gcc builtin just work for mingw (that
Not sure about this. I don't have a mingw environment to test, but I trust gcc and guess it does.
is, no need to use the OS-specific InterlockedIncrement if you have the compiler builtin instead).
I think this file needs three implementations:
#if defined __GNUC__ || <detect Intel's compiler, which also has these> use compiler builtins of __sync_add_and_fetch #elif defined WIN32 use OS primitives, like InterlockedIncrement #else we're hosed when it comes to lightweight versions, but we can still implement a heavyweight replacement that uses virMutex #endif
Agreed, this is a better way.
+++ b/src/util/viratomic.h @@ -0,0 +1,30 @@
+#ifndef __VIR_ATOMIC_H +#define __VIR_ATOMIC_H + +long virAtomicInc(long *value); +long virAtomicDec(long *value);
Mark both of these ATTRIBUTE_NONNULL(1)
OK.
I'm debating whether they should also be marked ATTRIBUTE_RETURN_CHECK
No, there are cases you just want to inc/dec a value but do not check the modified value.
+++ b/src/util/virobject.c @@ -0,0 +1,52 @@ + +#include "viratomic.h" +#include "virobject.h" +#include "logging.h" + +int virObjectInit(virObjectPtr obj, void (*free)(virObjectPtr obj))
You should declare a typedef:
typedef void (*virObjectFreer)(virObjectPtr);
then it becomes simpler to read:
int virObjectInit(virObjectPtr obj, virObjectFreer f)
Use a different name (I used freer/f above) to avoid -Wshadow warnings with free().
OK.
+{ + if (!free) {
Especially since shadowing means it's impossible to tell if this statement is always true (the address of free() exists) or conditional (there is a local variable named free shadowing the global function), and context-sensitive code reviews are tougher :)
+ VIR_ERROR0("method free is required."); + return -1; + }
Should this function also check and return -1 if obj->free was not NULL on entry (that is, guarantee that you can't initialize an object twice)?
Agreed. And it is dangerous to initialize an object more than once because the ref count may already changed and a second initializaton just do the wrong thing.
+ + obj->ref = 1; + obj->free = free; + + return 0; +} + +void virObjectRef(virObjectPtr obj) +{ + sa_assert(obj->ref > 0);
This is useless. It only helps static analyzers (like clang),
+ virAtomicInc(&obj->ref);
but there's nothing to analyze that depends on knowing the value was positive.
I'm debating whether to do checking. Maybe we should do:
if (virAtomicInc(&obj->ref) < 2)
This means bad things happened, and it is not enough to just give a warning. Think about this: two threads visit an object whose memory is corrupted that it's ref count becomes 0 but it doesn't get freed. thread 1: thread 2: ref the object (ref count becomes 1) unref the object (ref count becomes 0, object being freed) do something with the object, but it is already freed by thread 2 We should catch abnormal ref count by some way(if not sa_assert)
VIR_WARN("invalid call to virObjectRef");
+void virObjectUnref(virObjectPtr obj) +{ + sa_assert(obj->ref > 0);
Again, I'm not sure that this buys anything. But we may want to do:
int ref = virAtomicDec(&obj->ref); if (ref < 0) VIR_WARN("invalid call to virObjectUnref"); else if (ref == 0) obj->free(obj)
+ if (virAtomicDec(&obj->ref) == 0) + obj->free(obj); +} diff --git a/src/util/virobject.h b/src/util/virobject.h new file mode 100644 index 0000000..cd7d3e8 --- /dev/null +++ b/src/util/virobject.h
+ +typedef struct _virObject virObject; +typedef virObject *virObjectPtr; + +struct _virObject { + long ref; + void (*free)(virObjectPtr obj);
Is virObjectPtr the right thing to use here? If we assume that all clients will always have a virObjectPtr as their first member, then they can cast that back into the larger object. Is void* opaque any better,
Exactly the way(you have already saw my other patches when I was typing this:))
although that then it requires a third parameter for virObjectInit? Maybe it's worth some documentation on intended use in this header (am I
OK.
getting this usage right? I haven't looked at how you used it later in the series, but am just guessing):
struct _virFoo { virObject obj; /* Must be first member */ ... }; static void virFooFree(virObjectPtr obj) { virFooPtr foo = obj; ... } virFooPtr virFooNew(void) { virFooPtr foo; if (VIR_ALLOC(foo) < 0) { virReportOOMError(); return NULL; } virObjectInit(&foo->obj, virFooFree); ... return foo; }
+}; + +int virObjectInit(virObjectPtr obj, void (*free)(virObjectPtr obj));
ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) ATTRIBUTE_RETURN_CHECK
+void virObjectRef(virObjectPtr obj);
ATTRIBUTE_NONNULL(1)
Also, should this return the current reference count? Not all callers will need it, but returning void is harsh if someone might be able to use it.
No. It is meaningless to return the current reference count, because it may already changed by others after the caller reads the returned value on which it is not safe to make some choice.
+void virObjectUnref(virObjectPtr obj);
Likewise.
-- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

On Thu, Apr 07, 2011 at 03:49:15PM +0800, Hu Tao wrote:
On Wed, Apr 06, 2011 at 02:19:57PM -0600, Eric Blake wrote:
On 04/06/2011 01:19 AM, Hu Tao wrote:
virObject is the base struct that manages reference-counting for all structs that need the ability of reference-counting.
virAtomic provides atomic operations which are thread-safe. --- src/Makefile.am | 2 + src/libvirt_private.syms | 5 ++++ src/util/viratomic.c | 46 ++++++++++++++++++++++++++++++++++++++++ src/util/viratomic.h | 30 ++++++++++++++++++++++++++ src/util/virobject.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++ src/util/virobject.h | 39 ++++++++++++++++++++++++++++++++++ 6 files changed, 174 insertions(+), 0 deletions(-) create mode 100644 src/util/viratomic.c create mode 100644 src/util/viratomic.h create mode 100644 src/util/virobject.c create mode 100644 src/util/virobject.h
+++ b/src/libvirt_private.syms @@ -993,3 +993,8 @@ virXPathUInt; virXPathULong; virXPathULongHex; virXPathULongLong; + +# object.h
virobject.h, and float this section to appear before virtaudit.h (hmm,
Oops, there are always things escape from under your nose:(
maybe we should do a preliminary patch to rename that to viraudit.h, so that we aren't mixing vir vs. virt in quite so many places).
+virObjectInit; +virObjectRef; +virObjectUnref;
Missing exports from viratomic.h.
Why the linker doesn't complain about this this time?
diff --git a/src/util/viratomic.c b/src/util/viratomic.c new file mode 100644 index 0000000..629f189 --- /dev/null +++ b/src/util/viratomic.c @@ -0,0 +1,46 @@
+ +#ifdef WIN32 +long virAtomicInc(long *value) +{ + return InterlockedIncrement(value); +} + +long virAtomicDec(long *value) +{ + return InterlockedDecrement(value);
This is an OS-specific replacement.
+} +#else /* WIN32 */ +long virAtomicInc(long *value) +{ + return __sync_add_and_fetch(value, 1);
This is a gcc builtin, and will fail to compile with other C99
Yes.
compilers. Meanwhile, won't the gcc builtin just work for mingw (that
Not sure about this. I don't have a mingw environment to test, but I trust gcc and guess it does.
is, no need to use the OS-specific InterlockedIncrement if you have the compiler builtin instead).
I think this file needs three implementations:
#if defined __GNUC__ || <detect Intel's compiler, which also has these> use compiler builtins of __sync_add_and_fetch #elif defined WIN32 use OS primitives, like InterlockedIncrement #else we're hosed when it comes to lightweight versions, but we can still implement a heavyweight replacement that uses virMutex #endif
Agreed, this is a better way.
+++ b/src/util/viratomic.h @@ -0,0 +1,30 @@
+#ifndef __VIR_ATOMIC_H +#define __VIR_ATOMIC_H + +long virAtomicInc(long *value); +long virAtomicDec(long *value);
Mark both of these ATTRIBUTE_NONNULL(1)
OK.
I'm debating whether they should also be marked ATTRIBUTE_RETURN_CHECK
No, there are cases you just want to inc/dec a value but do not check the modified value.
+++ b/src/util/virobject.c @@ -0,0 +1,52 @@ + +#include "viratomic.h" +#include "virobject.h" +#include "logging.h" + +int virObjectInit(virObjectPtr obj, void (*free)(virObjectPtr obj))
You should declare a typedef:
typedef void (*virObjectFreer)(virObjectPtr);
then it becomes simpler to read:
int virObjectInit(virObjectPtr obj, virObjectFreer f)
Use a different name (I used freer/f above) to avoid -Wshadow warnings with free().
OK.
+{ + if (!free) {
Especially since shadowing means it's impossible to tell if this statement is always true (the address of free() exists) or conditional (there is a local variable named free shadowing the global function), and context-sensitive code reviews are tougher :)
+ VIR_ERROR0("method free is required."); + return -1; + }
Should this function also check and return -1 if obj->free was not NULL on entry (that is, guarantee that you can't initialize an object twice)?
Agreed. And it is dangerous to initialize an object more than once because the ref count may already changed and a second initializaton just do the wrong thing.
+ + obj->ref = 1; + obj->free = free; + + return 0; +} + +void virObjectRef(virObjectPtr obj) +{ + sa_assert(obj->ref > 0);
This is useless. It only helps static analyzers (like clang),
+ virAtomicInc(&obj->ref);
but there's nothing to analyze that depends on knowing the value was positive.
I'm debating whether to do checking. Maybe we should do:
if (virAtomicInc(&obj->ref) < 2)
This means bad things happened, and it is not enough to just give a warning. Think about this:
two threads visit an object whose memory is corrupted that it's ref count becomes 0 but it doesn't get freed.
thread 1: thread 2:
ref the object (ref count becomes 1)
unref the object (ref count becomes 0, object being freed)
do something with the object, but it is already freed by thread 2
We should catch abnormal ref count by some way(if not sa_assert)
VIR_WARN("invalid call to virObjectRef");
+void virObjectUnref(virObjectPtr obj) +{ + sa_assert(obj->ref > 0);
Again, I'm not sure that this buys anything. But we may want to do:
int ref = virAtomicDec(&obj->ref); if (ref < 0) VIR_WARN("invalid call to virObjectUnref"); else if (ref == 0) obj->free(obj)
+ if (virAtomicDec(&obj->ref) == 0) + obj->free(obj); +} diff --git a/src/util/virobject.h b/src/util/virobject.h new file mode 100644 index 0000000..cd7d3e8 --- /dev/null +++ b/src/util/virobject.h
+ +typedef struct _virObject virObject; +typedef virObject *virObjectPtr; + +struct _virObject { + long ref; + void (*free)(virObjectPtr obj);
Is virObjectPtr the right thing to use here? If we assume that all clients will always have a virObjectPtr as their first member, then they can cast that back into the larger object. Is void* opaque any better,
Exactly the way(you have already saw my other patches when I was typing this:))
although that then it requires a third parameter for virObjectInit? Maybe it's worth some documentation on intended use in this header (am I
OK.
getting this usage right? I haven't looked at how you used it later in the series, but am just guessing):
struct _virFoo { virObject obj; /* Must be first member */ ... }; static void virFooFree(virObjectPtr obj) { virFooPtr foo = obj; ... } virFooPtr virFooNew(void) { virFooPtr foo; if (VIR_ALLOC(foo) < 0) { virReportOOMError(); return NULL; } virObjectInit(&foo->obj, virFooFree); ... return foo; }
+}; + +int virObjectInit(virObjectPtr obj, void (*free)(virObjectPtr obj));
ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) ATTRIBUTE_RETURN_CHECK
+void virObjectRef(virObjectPtr obj);
ATTRIBUTE_NONNULL(1)
Also, should this return the current reference count? Not all callers will need it, but returning void is harsh if someone might be able to use it.
No. It is meaningless to return the current reference count, because it may already changed by others after the caller reads the returned value on which it is not safe to make some choice.
I wondered if you had an updated copy of this single patch, with the changes from this discussion included ? Even though we're still debating the rest of this patch series, I'd like us to commit this first virAtomic / virObject patch, so I can use it in some other code I have waiting. So if you are able to re-post just this first virObject patch that would be very helpful Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

This prepares for the next patch. The bad is we have no way to check the return value for CreateMutex when it is used as a static initializer. --- src/util/threads-pthread.h | 5 +++++ src/util/threads-win32.h | 5 +++++ 2 files changed, 10 insertions(+), 0 deletions(-) diff --git a/src/util/threads-pthread.h b/src/util/threads-pthread.h index b25d0c2..ff50253 100644 --- a/src/util/threads-pthread.h +++ b/src/util/threads-pthread.h @@ -23,6 +23,11 @@ #include <pthread.h> +#define VIR_MUTEX_INITIALIZER \ +{ \ + .lock = PTHREAD_MUTEX_INITIALIZER \ +} + struct virMutex { pthread_mutex_t lock; }; diff --git a/src/util/threads-win32.h b/src/util/threads-win32.h index bb7c455..cfadbe4 100644 --- a/src/util/threads-win32.h +++ b/src/util/threads-win32.h @@ -23,6 +23,11 @@ #include <windows.h> +#define VIR_MUTEX_INITIALIZER \ +{ \ + .lock = CreateMutex(NULL, FALSE, NULL) \ +} + struct virMutex { HANDLE lock; }; -- 1.7.3.1

virObject is the base struct that manages reference-counting for all structs that need the ability of reference-counting. virAtomic provides atomic operations which are thread-safe. --- src/Makefile.am | 2 + src/libvirt_private.syms | 11 ++++++ src/util/viratomic.c | 80 ++++++++++++++++++++++++++++++++++++++++++ src/util/viratomic.h | 32 +++++++++++++++++ src/util/virobject.c | 64 ++++++++++++++++++++++++++++++++++ src/util/virobject.h | 86 ++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 275 insertions(+), 0 deletions(-) create mode 100644 src/util/viratomic.c create mode 100644 src/util/viratomic.h create mode 100644 src/util/virobject.c create mode 100644 src/util/virobject.h diff --git a/src/Makefile.am b/src/Makefile.am index dce866e..cb8acc3 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -81,6 +81,8 @@ UTIL_SOURCES = \ util/util.c util/util.h \ util/xml.c util/xml.h \ util/virtaudit.c util/virtaudit.h \ + util/viratomic.c util/viratomic.h \ + util/virobject.c util/virobject.h \ util/virterror.c util/virterror_internal.h EXTRA_DIST += util/threads-pthread.c util/threads-win32.c diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index 54e4482..a33e877 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -958,6 +958,17 @@ virUUIDGenerate; virUUIDParse; +# virobject.h +virObjectInit; +virObjectRef; +virObjectUnref; + + +# viratomic.h +virAtomicInc; +virAtomicDec; + + # virtaudit.h virAuditClose; virAuditEncode; diff --git a/src/util/viratomic.c b/src/util/viratomic.c new file mode 100644 index 0000000..c7bb44f --- /dev/null +++ b/src/util/viratomic.c @@ -0,0 +1,80 @@ +/* + * viratomic.c: atomic operations + * + * Copyright (C) 2011 Hu Tao + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Authors: + * Hu Tao <hutao@cn.fujitsu.com> + */ + +#include <config.h> + +#include "viratomic.h" + +#if defined(__GNUC__) + +long virAtomicInc(long *value) +{ + return __sync_add_and_fetch(value, 1); +} + +long virAtomicDec(long *value) +{ + return __sync_sub_and_fetch(value, 1); +} + +#elif defined(WIN32) + +long virAtomicInc(long *value) +{ + return InterlockedIncrement(value); +} + +long virAtomicDec(long *value) +{ + return InterlockedDecrement(value); +} + +#else + +#include "threads.h" + +static virMutex virAtomicLock = VIR_MUTEX_INITIALIZER; + +long virAtomicInc(long *value) +{ + long result; + virMutexLock(&virAtomicLock); + *value += 1; + result = *value; + virMutexUnlock(&virAtomicLock); + + return result; +} + +long virAtomicDec(long *value) +{ + long result; + virMutexLock(&virAtomicLock); + *value -= 1; + result = *value; + virMutexUnlock(&virAtomicLock); + + return result; +} + +#endif diff --git a/src/util/viratomic.h b/src/util/viratomic.h new file mode 100644 index 0000000..2b40c77 --- /dev/null +++ b/src/util/viratomic.h @@ -0,0 +1,32 @@ +/* + * viratomic.h: atomic operations + * + * Copyright (C) 2011 Hu Tao + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Authors: + * Hu Tao <hutao@cn.fujitsu.com> + */ + +#ifndef __VIR_ATOMIC_H +#define __VIR_ATOMIC_H + +#include "internal.h" + +long virAtomicInc(long *value) ATTRIBUTE_NONNULL(1); +long virAtomicDec(long *value) ATTRIBUTE_NONNULL(1); + +#endif /* __VIR_ATOMIC_H */ diff --git a/src/util/virobject.c b/src/util/virobject.c new file mode 100644 index 0000000..0af53e9 --- /dev/null +++ b/src/util/virobject.c @@ -0,0 +1,64 @@ +/* + * virobject.c: base object that manages reference-counting + * + * Copyright (C) 2011 Hu Tao + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Authors: + * Hu Tao <hutao@cn.fujitsu.com> + */ + +#include <stdlib.h> + +#include "viratomic.h" +#include "virobject.h" +#include "logging.h" + +int virObjectInit(virObjectPtr obj, virObjectFree f) +{ + if (obj->free) { + /* This means the obj is already initialized. A second + * initialization will destroy ref field, which is bad. + */ + VIR_ERROR("Object %p is already initialized.", obj); + return -1; + } + + if (!f) { + VIR_ERROR0("virObjectFree method is required."); + abort(); + } + + obj->ref = 1; + obj->free = f; + + return 0; +} + +void virObjectRef(virObjectPtr obj) +{ + if (virAtomicInc(&obj->ref) < 2) + abort(); /* BUG if we get here */ +} + +void virObjectUnref(virObjectPtr obj) +{ + long ref = virAtomicDec(&obj->ref); + if (ref == 0) + obj->free(obj); + else if (ref < 0) + abort(); /* BUG if we get here */ +} diff --git a/src/util/virobject.h b/src/util/virobject.h new file mode 100644 index 0000000..a9d3d35 --- /dev/null +++ b/src/util/virobject.h @@ -0,0 +1,86 @@ +/* + * virobject.h: base object that manages reference-counting + * + * Copyright (C) 2011 Hu Tao + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Authors: + * Hu Tao <hutao@cn.fujitsu.com> + */ + +#ifndef __VIR_OBJECT_H +#define __VIR_OBJECT_H + +typedef struct _virObject virObject; +typedef virObject *virObjectPtr; + +typedef void (*virObjectFree)(virObjectPtr obj); + +/* A thread owns a virObject by incrementing its reference-count by 1. + * If a thread owns a virObject, the virObject is guarenteed not be + * freed until the thread releases ownership by decrementing its + * reference-count by 1. A thread can't access a virObject after it + * releases the ownership of virObject because it can be freed at + * anytime. + * + * A thread can own a virObject legally in these ways: + * + * - a thread owns a virObject that it creates. + * - a thread owns a virObject if another thread passes the ownership + * to it. Example: qemuMonitorOpen + * - a thread gets a virObject from a container. + * Example: virDomainFindByUUID + * - a container owns a virObject by incrementing its reference-count + * by 1 before adding it to the container + * - if a virObject is removed from a container its reference-count + * must be decremented by 1 + */ +/* + * Usage of virObject: + * + * 1. embed virObject into your struct as the first member. + * + * struct foo { + * virObject obj; // must be the first member + * ... + * } + * + * 2. call virObjectInit(obj, fooFree) in your struct's initialization + * function. fooFree is the free method which be called automatically + * when the ref count becomes 0. + * + * 3. implement two functions to ref/unref your struct like this: + * + * void FooRef(struct foo *f) + * { + * virObjectRef(f->obj); + * } + * + * void FooUnref(struct foo *f) + * { + * virObjectUnref(f->obj); + * } + */ +struct _virObject { + long ref; + virObjectFree free; +}; + +int virObjectInit(virObjectPtr obj, virObjectFree f); +void virObjectRef(virObjectPtr obj); +void virObjectUnref(virObjectPtr obj); + +#endif /* __VIR_OBJECT_H */ -- 1.7.3.1 -- Thanks, Hu Tao

On 04/20/2011 03:15 AM, Hu Tao wrote:
virObject is the base struct that manages reference-counting for all structs that need the ability of reference-counting.
virAtomic provides atomic operations which are thread-safe. --- src/Makefile.am | 2 + src/libvirt_private.syms | 11 ++++++ src/util/viratomic.c | 80 ++++++++++++++++++++++++++++++++++++++++++ src/util/viratomic.h | 32 +++++++++++++++++ src/util/virobject.c | 64 ++++++++++++++++++++++++++++++++++ src/util/virobject.h | 86 ++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 275 insertions(+), 0 deletions(-) create mode 100644 src/util/viratomic.c create mode 100644 src/util/viratomic.h create mode 100644 src/util/virobject.c create mode 100644 src/util/virobject.h
+++ b/src/libvirt_private.syms @@ -958,6 +958,17 @@ virUUIDGenerate; virUUIDParse;
+# virobject.h +virObjectInit; +virObjectRef; +virObjectUnref; + + +# viratomic.h +virAtomicInc; +virAtomicDec;
Swap these two sections (chunks are alphabetic by header), as well as swap these two lines (within a header, lines are alphabetic).
+++ b/src/util/viratomic.c
+#else + +#include "threads.h" + +static virMutex virAtomicLock = VIR_MUTEX_INITIALIZER;
Hmm, if I implement virOnce first, maybe we should use that instead.
+ +long virAtomicInc(long *value) +{ + long result; + virMutexLock(&virAtomicLock); + *value += 1; + result = *value; + virMutexUnlock(&virAtomicLock);
Clunky, but does the job ;)
+#include <stdlib.h> + +#include "viratomic.h" +#include "virobject.h" +#include "logging.h" + +int virObjectInit(virObjectPtr obj, virObjectFree f) +{ + if (obj->free) { + /* This means the obj is already initialized. A second + * initialization will destroy ref field, which is bad. + */ + VIR_ERROR("Object %p is already initialized.", obj); + return -1; + } + + if (!f) { + VIR_ERROR0("virObjectFree method is required."); + abort();
Either both failures should abort(), or both should return -1. They are both programming errors, so I'm personally okay with abort(), but this would be the first use of abort() outside gnulib files, so I'll see if either Dan has opinions as well. In fact, if we go with abort() in both places, then this can return void instead of int. Likewise, by adding ATTRIBUTE_NONNULL(2) to the prototype, then the compiler (well, at least clang) will help catch coding errors of failing to pass a callback function. Oops - you used TAB instead of space; 'make syntax-check' should have caught that.
+ +void virObjectRef(virObjectPtr obj) +{ + if (virAtomicInc(&obj->ref) < 2) + abort(); /* BUG if we get here */
If we keep the abort, then we _need_ the VIR_ERROR before hand saying why we quit.
+} + +void virObjectUnref(virObjectPtr obj) +{ + long ref = virAtomicDec(&obj->ref); + if (ref == 0) + obj->free(obj); + else if (ref < 0) + abort(); /* BUG if we get here */
Again, don't ever abort() without VIR_ERROR stating why. And maybe we should think about installing a SIGABRT handler so that aborts print a nicer message for the end user?
+ +typedef struct _virObject virObject; +typedef virObject *virObjectPtr; + +typedef void (*virObjectFree)(virObjectPtr obj);
I dislike extra casts, and this signature forces every client to have an explicit cast in their free function. Could we instead make this: typedef void (*virObjectFree)(void *obj); with some additional fallout documented below? Also, can we attach ATTRIBUTE_NONNULL(1) to a typedef? (I'm not quite sure how that works with gcc).
+ +/* A thread owns a virObject by incrementing its reference-count by 1. + * If a thread owns a virObject, the virObject is guarenteed not be
s/guarenteed/guaranteed to/
+ * freed until the thread releases ownership by decrementing its + * reference-count by 1. A thread can't access a virObject after it + * releases the ownership of virObject because it can be freed at + * anytime. + * + * A thread can own a virObject legally in these ways: + * + * - a thread owns a virObject that it creates. + * - a thread owns a virObject if another thread passes the ownership + * to it. Example: qemuMonitorOpen + * - a thread gets a virObject from a container. + * Example: virDomainFindByUUID + * - a container owns a virObject by incrementing its reference-count + * by 1 before adding it to the container + * - if a virObject is removed from a container its reference-count + * must be decremented by 1 + */ +/* + * Usage of virObject: + * + * 1. embed virObject into your struct as the first member. + * + * struct foo { + * virObject obj; // must be the first member + * ... + * } + * + * 2. call virObjectInit(obj, fooFree) in your struct's initialization + * function. fooFree is the free method which be called automatically + * when the ref count becomes 0.
The argument to fooFree will be &obj, which in turn can be cast as struct foo* because obj is the first member. struct foo * fooCreate() { struct foo *myfoo; if (VIR_ALLOC(myfoo) < 0) { virReportOOMError(); return NULL; } virObjectInit(&myfoo->obj, fooFree); ... // all other contents of myfoo return myfoo; }
+ * + * 3. implement two functions to ref/unref your struct like this: + * + * void FooRef(struct foo *f) + * { + * virObjectRef(f->obj); + * } + * + * void FooUnref(struct foo *f) + * { + * virObjectUnref(f->obj); + * }
4. implement the free method for your struct like this: static void fooFree(void *obj) { struct foo *myfoo = obj; ... // all other contents of myfoo VIR_FREE(myfoo); }
+ */ +struct _virObject { + long ref; + virObjectFree free; +}; + +int virObjectInit(virObjectPtr obj, virObjectFree f);
ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2)
+void virObjectRef(virObjectPtr obj);
ATTRIBUTE_NONNULL(1)
+void virObjectUnref(virObjectPtr obj);
ATTRIBUTE_NONNULL(1) -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

On 04/20/2011 03:14 AM, Hu Tao wrote:
This prepares for the next patch.
The bad is we have no way to check the return value for CreateMutex when it is used as a static initializer. --- src/util/threads-pthread.h | 5 +++++ src/util/threads-win32.h | 5 +++++ 2 files changed, 10 insertions(+), 0 deletions(-)
Then again, the next patch doesn't use VIR_MUTEX_INITIALIZER under WIN32 (it is only used in the fallback after gcc builtins and WIN32 native functions are bypassed). So, even simpler is just leaving it undefined for that platform, and deferring the problem of a working solution until the next time (if ever) we think we need the usage pattern of a static-initialized mutex to begin with. In other words,
diff --git a/src/util/threads-pthread.h b/src/util/threads-pthread.h index b25d0c2..ff50253 100644 --- a/src/util/threads-pthread.h +++ b/src/util/threads-pthread.h @@ -23,6 +23,11 @@
#include <pthread.h>
+#define VIR_MUTEX_INITIALIZER \ +{ \ + .lock = PTHREAD_MUTEX_INITIALIZER \ +} +
ACK to this hunk,
struct virMutex { pthread_mutex_t lock; }; diff --git a/src/util/threads-win32.h b/src/util/threads-win32.h index bb7c455..cfadbe4 100644 --- a/src/util/threads-win32.h +++ b/src/util/threads-win32.h @@ -23,6 +23,11 @@
#include <windows.h>
+#define VIR_MUTEX_INITIALIZER \ +{ \ + .lock = CreateMutex(NULL, FALSE, NULL) \ +} +
NAK to this hunk. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

On 04/20/2011 02:46 PM, Eric Blake wrote:
On 04/20/2011 03:14 AM, Hu Tao wrote:
This prepares for the next patch.
The bad is we have no way to check the return value for CreateMutex when it is used as a static initializer. --- src/util/threads-pthread.h | 5 +++++ src/util/threads-win32.h | 5 +++++ 2 files changed, 10 insertions(+), 0 deletions(-)
Then again, the next patch doesn't use VIR_MUTEX_INITIALIZER under WIN32 (it is only used in the fallback after gcc builtins and WIN32 native functions are bypassed). So, even simpler is just leaving it undefined for that platform, and deferring the problem of a working solution until the next time (if ever) we think we need the usage pattern of a static-initialized mutex to begin with.
In fact, I think that pthread_once/InitOnceExecuteOnce provides the perfect pthread vs. WIN32 approach to one-time initialization without relying on static initializers, and that it would be fairly easy to write virOnce as the wrapper function that covers the underlying functionality of both threading approaches. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

On 04/07/2011 01:49 AM, Hu Tao wrote:
+virObjectInit; +virObjectRef; +virObjectUnref;
Missing exports from viratomic.h.
Why the linker doesn't complain about this this time?
Because you didn't run configure (via ./autogen.sh) with the --with-driver-modules option. Missing exports don't bite until you are forcing dlopen() interaction between the various libraries, rather than linking them into one giant executable. (Which reminds me, I need to revive my patch to make ./autobuild.sh enable that option, to increase its test coverage). -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

--- daemon/dispatch.c | 2 - daemon/libvirtd.c | 62 +++++++++++++++++++++++++++++++++++++++------------- daemon/libvirtd.h | 5 ++- 3 files changed, 49 insertions(+), 20 deletions(-) diff --git a/daemon/dispatch.c b/daemon/dispatch.c index 4814017..b2d6927 100644 --- a/daemon/dispatch.c +++ b/daemon/dispatch.c @@ -524,9 +524,7 @@ remoteDispatchClientCall (struct qemud_server *server, */ rv = (data->fn)(server, client, conn, &msg->hdr, &rerr, &args, &ret); - virMutexLock(&server->lock); virMutexLock(&client->lock); - virMutexUnlock(&server->lock); xdr_free (data->args_filter, (char*)&args); diff --git a/daemon/libvirtd.c b/daemon/libvirtd.c index 42cbe5d..d3ba97b 100644 --- a/daemon/libvirtd.c +++ b/daemon/libvirtd.c @@ -270,6 +270,8 @@ static void sig_fatal(int sig, siginfo_t * siginfo ATTRIBUTE_UNUSED, static void qemudDispatchClientEvent(int watch, int fd, int events, void *opaque); static void qemudDispatchServerEvent(int watch, int fd, int events, void *opaque); static int qemudStartWorker(struct qemud_server *server, struct qemud_worker *worker); +static void qemudClientRef(struct qemud_client *client); +static void qemudClientUnref(struct qemud_client *client); void qemudClientMessageQueuePush(struct qemud_client_message **queue, @@ -1338,6 +1340,8 @@ int qemudGetSocketIdentity(int fd, uid_t *uid, pid_t *pid) { #endif +static void qemudFreeClient(virObjectPtr obj); + static int qemudDispatchServer(struct qemud_server *server, struct qemud_socket *sock) { int fd; virSocketAddr addr; @@ -1411,6 +1415,10 @@ static int qemudDispatchServer(struct qemud_server *server, struct qemud_socket VIR_ERROR0(_("cannot initialize mutex")); goto error; } + if (virObjectInit(&client->obj, qemudFreeClient)) { + VIR_ERROR0(_("cannot initialize virobject")); + goto error; + } client->magic = QEMUD_CLIENT_MAGIC; client->fd = fd; @@ -1576,12 +1584,10 @@ static struct qemud_client *qemudPendingJob(struct qemud_server *server) { int i; for (i = 0 ; i < server->nclients ; i++) { - virMutexLock(&server->clients[i]->lock); if (server->clients[i]->dx) { - /* Delibrately don't unlock client - caller wants the lock */ + qemudClientRef(server->clients[i]); return server->clients[i]; } - virMutexUnlock(&server->clients[i]->lock); } return NULL; } @@ -1604,31 +1610,36 @@ static void *qemudWorker(void *data) } } if (worker->quitRequest) { - virMutexUnlock(&client->lock); virMutexUnlock(&server->lock); + qemudClientUnref(client); return NULL; } worker->processingCall = 1; virMutexUnlock(&server->lock); - /* We own a locked client now... */ - client->refs++; + virMutexLock(&client->lock); /* Remove our message from dispatch queue while we use it */ msg = qemudClientMessageQueueServe(&client->dx); + if (!msg) { + virMutexUnlock(&client->lock); + qemudClientUnref(client); + continue; + } + /* This function drops the lock during dispatch, * and re-acquires it before returning */ if (remoteDispatchClientRequest (server, client, msg) < 0) { VIR_FREE(msg); qemudDispatchClientFailure(client); - client->refs--; virMutexUnlock(&client->lock); + qemudClientUnref(client); continue; } - client->refs--; virMutexUnlock(&client->lock); + qemudClientUnref(client); virMutexLock(&server->lock); worker->processingCall = 0; @@ -2155,6 +2166,7 @@ qemudDispatchClientEvent(int watch, int fd, int events, void *opaque) { virMutexLock(&server->clients[i]->lock); if (server->clients[i]->watch == watch) { client = server->clients[i]; + qemudClientRef(client); break; } virMutexUnlock(&server->clients[i]->lock); @@ -2168,6 +2180,7 @@ qemudDispatchClientEvent(int watch, int fd, int events, void *opaque) { if (client->fd != fd) { virMutexUnlock(&client->lock); + qemudClientUnref(client); return; } @@ -2190,6 +2203,7 @@ qemudDispatchClientEvent(int watch, int fd, int events, void *opaque) { qemudDispatchClientFailure(client); virMutexUnlock(&client->lock); + qemudClientUnref(client); } @@ -2306,7 +2320,9 @@ static void qemudInactiveTimer(int timerid, void *data) { } } -static void qemudFreeClient(struct qemud_client *client) { +static void qemudFreeClient(virObjectPtr obj) { + struct qemud_client *client = (struct qemud_client *)obj; + while (client->rx) { struct qemud_client_message *msg = qemudClientMessageQueueServe(&client->rx); @@ -2333,6 +2349,16 @@ static void qemudFreeClient(struct qemud_client *client) { VIR_FREE(client); } +static void qemudClientRef(struct qemud_client *client) +{ + virObjectRef(&client->obj); +} + +static void qemudClientUnref(struct qemud_client *client) +{ + virObjectUnref(&client->obj); +} + static void *qemudRunLoop(void *opaque) { struct qemud_server *server = opaque; int timerid = -1; @@ -2396,13 +2422,17 @@ static void *qemudRunLoop(void *opaque) { reprocess: for (i = 0 ; i < server->nclients ; i++) { - int inactive; + struct qemud_client *client = NULL; virMutexLock(&server->clients[i]->lock); - inactive = server->clients[i]->fd == -1 - && server->clients[i]->refs == 0; - virMutexUnlock(&server->clients[i]->lock); - if (inactive) { - qemudFreeClient(server->clients[i]); + if (server->clients[i]->fd == -1) { + client = server->clients[i]; + server->clients[i] = NULL; + } + if (!client) { + virMutexUnlock(&server->clients[i]->lock); + } else { + virMutexUnlock(&client->lock); + qemudClientUnref(client); server->nclients--; if (i < server->nclients) memmove(server->clients + i, @@ -2451,7 +2481,7 @@ cleanup: } VIR_FREE(server->workers); for (i = 0; i < server->nclients; i++) - qemudFreeClient(server->clients[i]); + qemudClientUnref(server->clients[i]); server->nclients = 0; VIR_SHRINK_N(server->clients, server->nclients_max, server->nclients_max); diff --git a/daemon/libvirtd.h b/daemon/libvirtd.h index d37c3fd..5ac4eb3 100644 --- a/daemon/libvirtd.h +++ b/daemon/libvirtd.h @@ -49,6 +49,7 @@ # include "logging.h" # include "threads.h" # include "network.h" +# include "virobject.h" # if WITH_DTRACE # ifndef LIBVIRTD_PROBES_H @@ -186,6 +187,8 @@ struct qemud_client_stream { /* Stores the per-client connection state */ struct qemud_client { + virObject obj; /* should be the first member */ + virMutex lock; int magic; @@ -243,8 +246,6 @@ struct qemud_client { * called, it will be set back to NULL if that succeeds. */ virConnectPtr conn; - int refs; - }; # define QEMUD_CLIENT_MAGIC 0x7788aaee -- 1.7.3.1 -- Thanks, Hu Tao

On 04/06/2011 01:19 AM, Hu Tao wrote:
--- daemon/dispatch.c | 2 - daemon/libvirtd.c | 62 +++++++++++++++++++++++++++++++++++++++------------- daemon/libvirtd.h | 5 ++- 3 files changed, 49 insertions(+), 20 deletions(-)
Looks like a sane conversion to me, once patch 1/6 is in good shape. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

This patch also eliminates a dead-lock bug in qemuDomainObjBeginJobWithDriver: if virCondWaitUntil() timeouts, the thread tries to acquire qemu driver lock while holding virDomainObj lock. --- src/conf/domain_conf.c | 56 ++++---- src/conf/domain_conf.h | 6 +- src/openvz/openvz_conf.c | 8 +- src/qemu/qemu_domain.c | 32 ++--- src/qemu/qemu_domain.h | 2 +- src/qemu/qemu_driver.c | 304 ++++++++++++++++++++------------------------- src/qemu/qemu_migration.c | 45 +++---- src/qemu/qemu_process.c | 33 ++--- src/vmware/vmware_conf.c | 2 +- 9 files changed, 215 insertions(+), 273 deletions(-) diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index 90a1317..fc76a00 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -47,6 +47,7 @@ #include "storage_file.h" #include "files.h" #include "bitmap.h" +#include "virobject.h" #define VIR_FROM_THIS VIR_FROM_DOMAIN @@ -395,9 +396,7 @@ static void virDomainObjListDataFree(void *payload, const void *name ATTRIBUTE_UNUSED) { virDomainObjPtr obj = payload; - virDomainObjLock(obj); - if (virDomainObjUnref(obj) > 0) - virDomainObjUnlock(obj); + virDomainObjUnref(obj); } int virDomainObjListInit(virDomainObjListPtr doms) @@ -437,7 +436,7 @@ virDomainObjPtr virDomainFindByID(const virDomainObjListPtr doms, virDomainObjPtr obj; obj = virHashSearch(doms->objs, virDomainObjListSearchID, &id); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj); return obj; } @@ -452,7 +451,7 @@ virDomainObjPtr virDomainFindByUUID(const virDomainObjListPtr doms, obj = virHashLookup(doms->objs, uuidstr); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj); return obj; } @@ -476,7 +475,7 @@ virDomainObjPtr virDomainFindByName(const virDomainObjListPtr doms, virDomainObjPtr obj; obj = virHashSearch(doms->objs, virDomainObjListSearchName, name); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj); return obj; } @@ -967,6 +966,12 @@ static void virDomainObjFree(virDomainObjPtr dom) { if (!dom) return; + virDomainObjUnref(dom); +} + +static void doDomainObjFree(virObjectPtr obj) +{ + virDomainObjPtr dom = (virDomainObjPtr)obj; VIR_DEBUG("obj=%p", dom); virDomainDefFree(dom->def); @@ -984,21 +989,13 @@ static void virDomainObjFree(virDomainObjPtr dom) void virDomainObjRef(virDomainObjPtr dom) { - dom->refs++; - VIR_DEBUG("obj=%p refs=%d", dom, dom->refs); + virObjectRef(&dom->obj); } -int virDomainObjUnref(virDomainObjPtr dom) +void virDomainObjUnref(virDomainObjPtr dom) { - dom->refs--; - VIR_DEBUG("obj=%p refs=%d", dom, dom->refs); - if (dom->refs == 0) { - virDomainObjUnlock(dom); - virDomainObjFree(dom); - return 0; - } - return dom->refs; + virObjectUnref(&dom->obj); } static virDomainObjPtr virDomainObjNew(virCapsPtr caps) @@ -1010,6 +1007,11 @@ static virDomainObjPtr virDomainObjNew(virCapsPtr caps) return NULL; } + if (virObjectInit(&domain->obj, doDomainObjFree)) { + VIR_FREE(domain); + return NULL; + } + if (caps->privateDataAllocFunc && !(domain->privateData = (caps->privateDataAllocFunc)())) { virReportOOMError(); @@ -1027,9 +1029,7 @@ static virDomainObjPtr virDomainObjNew(virCapsPtr caps) return NULL; } - virDomainObjLock(domain); domain->state = VIR_DOMAIN_SHUTOFF; - domain->refs = 1; virDomainSnapshotObjListInit(&domain->snapshots); @@ -1075,8 +1075,10 @@ virDomainObjPtr virDomainAssignDef(virCapsPtr caps, domain->def = def; virUUIDFormat(def->uuid, uuidstr); + virDomainObjRef(domain); if (virHashAddEntry(doms->objs, uuidstr, domain) < 0) { - VIR_FREE(domain); + virDomainObjUnref(domain); + virDomainObjFree(domain); return NULL; } @@ -1149,9 +1151,7 @@ virDomainObjGetPersistentDef(virCapsPtr caps, } /* - * The caller must hold a lock on the driver owning 'doms', - * and must also have locked 'dom', to ensure no one else - * is either waiting for 'dom' or still usingn it + * The caller must hold a lock on the driver owning 'doms'. */ void virDomainRemoveInactive(virDomainObjListPtr doms, virDomainObjPtr dom) @@ -1159,9 +1159,8 @@ void virDomainRemoveInactive(virDomainObjListPtr doms, char uuidstr[VIR_UUID_STRING_BUFLEN]; virUUIDFormat(dom->def->uuid, uuidstr); - virDomainObjUnlock(dom); - virHashRemoveEntry(doms->objs, uuidstr); + virDomainObjUnref(dom); } @@ -6146,7 +6145,7 @@ static virDomainObjPtr virDomainObjParseXML(virCapsPtr caps, error: /* obj was never shared, so unref should return 0 */ - ignore_value(virDomainObjUnref(obj)); + virDomainObjUnref(obj); return NULL; } @@ -8449,10 +8448,12 @@ static virDomainObjPtr virDomainLoadConfig(virCapsPtr caps, if (!(dom = virDomainAssignDef(caps, doms, def, false))) goto error; + virDomainObjLock(dom); dom->autostart = autostart; if (notify) (*notify)(dom, newVM, opaque); + virDomainObjUnlock(dom); VIR_FREE(configFile); VIR_FREE(autostartLink); @@ -8501,9 +8502,8 @@ static virDomainObjPtr virDomainLoadStatus(virCapsPtr caps, return obj; error: - /* obj was never shared, so unref should return 0 */ if (obj) - ignore_value(virDomainObjUnref(obj)); + virDomainObjUnref(obj); VIR_FREE(statusFile); return NULL; } diff --git a/src/conf/domain_conf.h b/src/conf/domain_conf.h index 10e73cb..3218672 100644 --- a/src/conf/domain_conf.h +++ b/src/conf/domain_conf.h @@ -40,6 +40,7 @@ # include "nwfilter_conf.h" # include "macvtap.h" # include "sysinfo.h" +# include "virobject.h" /* Private component of virDomainXMLFlags */ typedef enum { @@ -1144,8 +1145,8 @@ struct _virDomainDef { typedef struct _virDomainObj virDomainObj; typedef virDomainObj *virDomainObjPtr; struct _virDomainObj { + virObject obj; virMutex lock; - int refs; int pid; int state; @@ -1226,8 +1227,7 @@ int virDomainDeviceInfoIterate(virDomainDefPtr def, void virDomainDefFree(virDomainDefPtr vm); void virDomainObjRef(virDomainObjPtr vm); -/* Returns 1 if the object was freed, 0 if more refs exist */ -int virDomainObjUnref(virDomainObjPtr vm) ATTRIBUTE_RETURN_CHECK; +void virDomainObjUnref(virDomainObjPtr vm); /* live == true means def describes an active domain (being migrated or * restored) as opposed to a new persistent configuration of the domain */ diff --git a/src/openvz/openvz_conf.c b/src/openvz/openvz_conf.c index 88cd4c8..c08ed3b 100644 --- a/src/openvz/openvz_conf.c +++ b/src/openvz/openvz_conf.c @@ -472,6 +472,11 @@ int openvzLoadDomains(struct openvz_driver *driver) { if (VIR_ALLOC(dom) < 0) goto no_memory; + if (virObjectInit(&dom->obj, NULL)) { + VIR_FREE(dom); + goto cleanup; + } + if (virMutexInit(&dom->lock) < 0) { openvzError(VIR_ERR_INTERNAL_ERROR, "%s", _("cannot initialize mutex")); @@ -489,7 +494,6 @@ int openvzLoadDomains(struct openvz_driver *driver) { else dom->state = VIR_DOMAIN_RUNNING; - dom->refs = 1; dom->pid = veid; dom->def->id = dom->state == VIR_DOMAIN_SHUTOFF ? -1 : veid; /* XXX OpenVZ doesn't appear to have concept of a transient domain */ @@ -554,7 +558,7 @@ int openvzLoadDomains(struct openvz_driver *driver) { VIR_FREE(outbuf); /* dom hasn't been shared yet, so unref should return 0 */ if (dom) - ignore_value(virDomainObjUnref(dom)); + virDomainObjUnref(dom); return -1; } diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..3a3c953 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -457,13 +457,13 @@ int qemuDomainObjBeginJob(virDomainObjPtr obj) } then = timeval_to_ms(now) + QEMU_JOB_WAIT_TIME; - virDomainObjRef(obj); + virDomainObjLock(obj); while (priv->jobActive) { if (virCondWaitUntil(&priv->jobCond, &obj->lock, then) < 0) { - /* Safe to ignore value since ref count was incremented above */ - ignore_value(virDomainObjUnref(obj)); - if (errno == ETIMEDOUT) + int err = errno; + virDomainObjUnlock(obj); + if (err == ETIMEDOUT) qemuReportError(VIR_ERR_OPERATION_TIMEOUT, "%s", _("cannot acquire state change lock")); else @@ -482,12 +482,10 @@ int qemuDomainObjBeginJob(virDomainObjPtr obj) } /* - * obj must be locked before calling, qemud_driver must be locked - * * This must be called by anything that will change the VM state * in any way, or anything that will use the QEMU monitor. */ -int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, +int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver ATTRIBUTE_UNUSED, virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; @@ -501,20 +499,18 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, } then = timeval_to_ms(now) + QEMU_JOB_WAIT_TIME; - virDomainObjRef(obj); - qemuDriverUnlock(driver); + virDomainObjLock(obj); while (priv->jobActive) { if (virCondWaitUntil(&priv->jobCond, &obj->lock, then) < 0) { - /* Safe to ignore value since ref count was incremented above */ - ignore_value(virDomainObjUnref(obj)); - if (errno == ETIMEDOUT) + int err = errno; + virDomainObjUnlock(obj); + if (err == ETIMEDOUT) qemuReportError(VIR_ERR_OPERATION_TIMEOUT, "%s", _("cannot acquire state change lock")); else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); - qemuDriverLock(driver); return -1; } } @@ -524,10 +520,6 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, priv->jobStart = timeval_to_ms(now); memset(&priv->jobInfo, 0, sizeof(priv->jobInfo)); - virDomainObjUnlock(obj); - qemuDriverLock(driver); - virDomainObjLock(obj); - return 0; } @@ -540,7 +532,7 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, * Returns remaining refcount on 'obj', maybe 0 to indicated it * was deleted */ -int qemuDomainObjEndJob(virDomainObjPtr obj) +void qemuDomainObjEndJob(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; @@ -551,7 +543,7 @@ int qemuDomainObjEndJob(virDomainObjPtr obj) memset(&priv->jobInfo, 0, sizeof(priv->jobInfo)); virCondSignal(&priv->jobCond); - return virDomainObjUnref(obj); + virDomainObjUnlock(obj); } @@ -655,7 +647,7 @@ void qemuDomainObjExitRemoteWithDriver(struct qemud_driver *driver, virDomainObjLock(obj); /* Safe to ignore value, since we incremented ref in * qemuDomainObjEnterRemoteWithDriver */ - ignore_value(virDomainObjUnref(obj)); + virDomainObjUnref(obj); } diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index 8258900..12fd21d 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -97,7 +97,7 @@ void qemuDomainSetNamespaceHooks(virCapsPtr caps); int qemuDomainObjBeginJob(virDomainObjPtr obj) ATTRIBUTE_RETURN_CHECK; int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, virDomainObjPtr obj) ATTRIBUTE_RETURN_CHECK; -int qemuDomainObjEndJob(virDomainObjPtr obj) ATTRIBUTE_RETURN_CHECK; +void qemuDomainObjEndJob(virDomainObjPtr obj); void qemuDomainObjEnterMonitor(virDomainObjPtr obj); void qemuDomainObjExitMonitor(virDomainObjPtr obj); void qemuDomainObjEnterMonitorWithDriver(struct qemud_driver *driver, diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 48fe266..db89402 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -139,7 +139,6 @@ qemuAutostartDomain(void *payload, const void *name ATTRIBUTE_UNUSED, void *opaq struct qemuAutostartData *data = opaque; virErrorPtr err; - virDomainObjLock(vm); virResetLastError(); if (qemuDomainObjBeginJobWithDriver(data->driver, vm) < 0) { err = virGetLastError(); @@ -156,12 +155,8 @@ qemuAutostartDomain(void *payload, const void *name ATTRIBUTE_UNUSED, void *opaq err ? err->message : _("unknown error")); } - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); } - - if (vm) - virDomainObjUnlock(vm); } @@ -1055,12 +1050,13 @@ static virDomainPtr qemudDomainLookupByID(virConnectPtr conn, goto cleanup; } + virDomainObjLock(vm); dom = virGetDomain(conn, vm->def->name, vm->def->uuid); if (dom) dom->id = vm->def->id; + virDomainObjUnlock(vm); + virDomainObjUnref(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); return dom; } @@ -1082,12 +1078,13 @@ static virDomainPtr qemudDomainLookupByUUID(virConnectPtr conn, goto cleanup; } + virDomainObjLock(vm); dom = virGetDomain(conn, vm->def->name, vm->def->uuid); if (dom) dom->id = vm->def->id; + virDomainObjUnlock(vm); + virDomainObjUnref(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); return dom; } @@ -1107,12 +1104,13 @@ static virDomainPtr qemudDomainLookupByName(virConnectPtr conn, goto cleanup; } + virDomainObjLock(vm); dom = virGetDomain(conn, vm->def->name, vm->def->uuid); if (dom) dom->id = vm->def->id; + virDomainObjUnlock(vm); + virDomainObjUnref(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); return dom; } @@ -1133,11 +1131,13 @@ static int qemuDomainIsActive(virDomainPtr dom) _("no domain with matching uuid '%s'"), uuidstr); goto cleanup; } + + virDomainObjLock(obj); ret = virDomainObjIsActive(obj); + virDomainObjUnlock(obj); + virDomainObjUnref(obj); cleanup: - if (obj) - virDomainObjUnlock(obj); return ret; } @@ -1157,11 +1157,13 @@ static int qemuDomainIsPersistent(virDomainPtr dom) _("no domain with matching uuid '%s'"), uuidstr); goto cleanup; } + virDomainObjLock(obj); ret = obj->persistent; + virDomainObjUnlock(obj); cleanup: if (obj) - virDomainObjUnlock(obj); + virDomainObjUnref(obj); return ret; } @@ -1181,11 +1183,13 @@ static int qemuDomainIsUpdated(virDomainPtr dom) _("no domain with matching uuid '%s'"), uuidstr); goto cleanup; } + virDomainObjLock(obj); ret = obj->updated; + virDomainObjUnlock(obj); cleanup: if (obj) - virDomainObjUnlock(obj); + virDomainObjUnref(obj); return ret; } @@ -1239,39 +1243,57 @@ static virDomainPtr qemudDomainCreate(virConnectPtr conn, const char *xml, qemuDriverLock(driver); if (!(def = virDomainDefParseString(driver->caps, xml, - VIR_DOMAIN_XML_INACTIVE))) + VIR_DOMAIN_XML_INACTIVE))) { + qemuDriverUnlock(driver); goto cleanup; + } - if (virSecurityManagerVerify(driver->securityManager, def) < 0) + if (virSecurityManagerVerify(driver->securityManager, def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + } - if (virDomainObjIsDuplicate(&driver->domains, def, 1) < 0) + if (virDomainObjIsDuplicate(&driver->domains, def, 1) < 0) { + qemuDriverUnlock(driver); goto cleanup; + } - if (qemudCanonicalizeMachine(driver, def) < 0) + if (qemudCanonicalizeMachine(driver, def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + } - if (qemuDomainAssignPCIAddresses(def) < 0) + if (qemuDomainAssignPCIAddresses(def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + } if (!(vm = virDomainAssignDef(driver->caps, &driver->domains, - def, false))) + def, false))) { + qemuDriverUnlock(driver); goto cleanup; + } + + qemuDriverUnlock(driver); def = NULL; - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) { + virDomainObjUnref(vm); goto cleanup; /* XXXX free the 'vm' we created ? */ + } if (qemuProcessStart(conn, driver, vm, NULL, (flags & VIR_DOMAIN_START_PAUSED) != 0, -1, NULL, VIR_VM_OP_CREATE) < 0) { qemuAuditDomainStart(vm, "booted", false); - if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, - vm); - vm = NULL; + qemuDomainObjEndJob(vm); + qemuDriverLock(driver); + virDomainRemoveInactive(&driver->domains, + vm); + qemuDriverUnlock(driver); + virDomainObjUnref(vm); goto cleanup; } @@ -1283,17 +1305,13 @@ static virDomainPtr qemudDomainCreate(virConnectPtr conn, const char *xml, dom = virGetDomain(conn, vm->def->name, vm->def->uuid); if (dom) dom->id = vm->def->id; - if (vm && - qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); + virDomainObjUnref(vm); cleanup: virDomainDefFree(def); - if (vm) - virDomainObjUnlock(vm); if (event) qemuDomainEventQueue(driver, event); - qemuDriverUnlock(driver); return dom; } @@ -1307,13 +1325,14 @@ static int qemudDomainSuspend(virDomainPtr dom) { qemuDriverLock(driver); vm = virDomainFindByUUID(&driver->domains, dom->uuid); + qemuDriverUnlock(driver); if (!vm) { char uuidstr[VIR_UUID_STRING_BUFLEN]; virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } if (!virDomainObjIsActive(vm)) { qemuReportError(VIR_ERR_OPERATION_INVALID, @@ -1354,16 +1373,12 @@ static int qemudDomainSuspend(virDomainPtr dom) { } endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); - + virDomainObjUnref(vm); if (event) qemuDomainEventQueue(driver, event); - qemuDriverUnlock(driver); return ret; } @@ -1376,6 +1391,7 @@ static int qemudDomainResume(virDomainPtr dom) { qemuDriverLock(driver); vm = virDomainFindByUUID(&driver->domains, dom->uuid); + qemuDriverUnlock(driver); if (!vm) { char uuidstr[VIR_UUID_STRING_BUFLEN]; @@ -1409,15 +1425,12 @@ static int qemudDomainResume(virDomainPtr dom) { ret = 0; endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); + virDomainObjUnref(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); if (event) qemuDomainEventQueue(driver, event); - qemuDriverUnlock(driver); return ret; } @@ -1437,7 +1450,7 @@ static int qemudDomainShutdown(virDomainPtr dom) { virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } if (qemuDomainObjBeginJob(vm) < 0) @@ -1455,12 +1468,10 @@ static int qemudDomainShutdown(virDomainPtr dom) { qemuDomainObjExitMonitor(vm); endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); + virDomainObjUnref(vm); return ret; } @@ -1473,12 +1484,13 @@ static int qemudDomainDestroy(virDomainPtr dom) { qemuDriverLock(driver); vm = virDomainFindByUUID(&driver->domains, dom->uuid); + qemuDriverUnlock(driver); if (!vm) { char uuidstr[VIR_UUID_STRING_BUFLEN]; virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) @@ -1497,24 +1509,24 @@ static int qemudDomainDestroy(virDomainPtr dom) { qemuAuditDomainStop(vm, "destroyed"); if (!vm->persistent) { - if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, - vm); + qemuDomainObjEndJob(vm); + virDomainObjUnref(vm); + qemuDriverLock(driver); + virDomainRemoveInactive(&driver->domains, vm); + qemuDriverUnlock(driver); vm = NULL; } ret = 0; endjob: - if (vm && - qemuDomainObjEndJob(vm) == 0) - vm = NULL; + if (vm) + qemuDomainObjEndJob(vm); cleanup: if (vm) - virDomainObjUnlock(vm); + virDomainObjUnref(vm); if (event) qemuDomainEventQueue(driver, event); - qemuDriverUnlock(driver); return ret; } @@ -1535,12 +1547,14 @@ static char *qemudDomainGetOSType(virDomainPtr dom) { goto cleanup; } + virDomainObjLock(vm); if (!(type = strdup(vm->def->os.type))) virReportOOMError(); + virDomainObjUnlock(vm); cleanup: if (vm) - virDomainObjUnlock(vm); + virDomainObjUnref(vm); return type; } @@ -1562,11 +1576,13 @@ static unsigned long qemudDomainGetMaxMemory(virDomainPtr dom) { goto cleanup; } + virDomainObjLock(vm); ret = vm->def->mem.max_balloon; + virDomainObjUnlock(vm); cleanup: if (vm) - virDomainObjUnlock(vm); + virDomainObjUnref(vm); return ret; } @@ -1594,18 +1610,18 @@ static int qemudDomainSetMemoryFlags(virDomainPtr dom, unsigned long newmem, virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } + if (qemuDomainObjBeginJob(vm) < 0) + goto cleanup; + if (newmem > vm->def->mem.max_balloon) { qemuReportError(VIR_ERR_INVALID_ARG, "%s", _("cannot set memory higher than max memory")); - goto cleanup; + goto endjob; } - if (qemuDomainObjBeginJob(vm) < 0) - goto cleanup; - if (!virDomainObjIsActive(vm) && (flags & VIR_DOMAIN_MEM_LIVE)) { qemuReportError(VIR_ERR_OPERATION_INVALID, "%s", _("domain is not running")); @@ -1647,12 +1663,10 @@ static int qemudDomainSetMemoryFlags(virDomainPtr dom, unsigned long newmem, ret = 0; endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); + virDomainObjUnref(vm); return ret; } @@ -1676,9 +1690,12 @@ static int qemudDomainGetInfo(virDomainPtr dom, virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } + if (qemuDomainObjBeginJob(vm) < 0) + goto cleanup; + info->state = vm->state; if (!virDomainObjIsActive(vm)) { @@ -1687,7 +1704,7 @@ static int qemudDomainGetInfo(virDomainPtr dom, if (qemudGetProcessInfo(&(info->cpuTime), NULL, vm->pid, 0) < 0) { qemuReportError(VIR_ERR_OPERATION_FAILED, "%s", _("cannot read cputime for domain")); - goto cleanup; + goto endjob; } } @@ -1700,8 +1717,6 @@ static int qemudDomainGetInfo(virDomainPtr dom, (vm->def->memballoon->model == VIR_DOMAIN_MEMBALLOON_MODEL_NONE)) { info->memory = vm->def->mem.max_balloon; } else if (!priv->jobActive) { - if (qemuDomainObjBeginJob(vm) < 0) - goto cleanup; if (!virDomainObjIsActive(vm)) err = 0; else { @@ -1709,10 +1724,6 @@ static int qemudDomainGetInfo(virDomainPtr dom, err = qemuMonitorGetBalloonInfo(priv->mon, &balloon); qemuDomainObjExitMonitor(vm); } - if (qemuDomainObjEndJob(vm) == 0) { - vm = NULL; - goto cleanup; - } if (err < 0) goto cleanup; @@ -1731,9 +1742,10 @@ static int qemudDomainGetInfo(virDomainPtr dom, info->nrVirtCpu = vm->def->vcpus; ret = 0; +endjob: + qemuDomainObjEndJob(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); + virDomainObjUnref(vm); return ret; } @@ -2006,9 +2018,8 @@ static int qemudDomainSaveFlag(struct qemud_driver *driver, virDomainPtr dom, VIR_DOMAIN_EVENT_STOPPED, VIR_DOMAIN_EVENT_STOPPED_SAVED); if (!vm->persistent) { - if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, - vm); + qemuDomainObjEndJob(vm); + virDomainRemoveInactive(&driver->domains, vm); vm = NULL; } @@ -2021,8 +2032,8 @@ endjob: VIR_WARN0("Unable to resume guest CPUs after save failure"); } } - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + + qemuDomainObjEndJob(vm); } cleanup: @@ -2303,6 +2314,7 @@ static int qemudDomainCoreDump(virDomainPtr dom, qemuDriverLock(driver); vm = virDomainFindByUUID(&driver->domains, dom->uuid); + qemuDriverUnlock(driver); if (!vm) { char uuidstr[VIR_UUID_STRING_BUFLEN]; @@ -2311,11 +2323,12 @@ static int qemudDomainCoreDump(virDomainPtr dom, _("no domain with matching uuid '%s'"), uuidstr); goto cleanup; } - priv = vm->privateData; if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) goto cleanup; + priv = vm->privateData; + if (!virDomainObjIsActive(vm)) { qemuReportError(VIR_ERR_OPERATION_INVALID, "%s", _("domain is not running")); @@ -2367,20 +2380,20 @@ endjob: } } - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; - else if ((ret == 0) && (flags & VIR_DUMP_CRASH) && !vm->persistent) { + qemuDomainObjEndJob(vm); + if ((ret == 0) && (flags & VIR_DUMP_CRASH) && !vm->persistent) { + qemuDriverLock(driver); virDomainRemoveInactive(&driver->domains, vm); + qemuDriverUnlock(driver); vm = NULL; } cleanup: - if (vm) - virDomainObjUnlock(vm); if (event) qemuDomainEventQueue(driver, event); - qemuDriverUnlock(driver); + if (vm) + virDomainObjUnref(vm); return ret; } @@ -2403,9 +2416,6 @@ static void processWatchdogEvent(void *data, void *opaque) break; } - qemuDriverLock(driver); - virDomainObjLock(wdEvent->vm); - if (qemuDomainObjBeginJobWithDriver(driver, wdEvent->vm) < 0) break; @@ -2429,10 +2439,7 @@ static void processWatchdogEvent(void *data, void *opaque) qemuReportError(VIR_ERR_OPERATION_FAILED, "%s", _("Resuming after dump failed")); - if (qemuDomainObjEndJob(wdEvent->vm) > 0) - virDomainObjUnlock(wdEvent->vm); - - qemuDriverUnlock(driver); + qemuDomainObjEndJob(wdEvent->vm); VIR_FREE(dumpfile); } @@ -2533,7 +2540,7 @@ qemudDomainSetVcpusFlags(virDomainPtr dom, unsigned int nvcpus, virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } if (qemuDomainObjBeginJob(vm) < 0) @@ -2608,12 +2615,10 @@ qemudDomainSetVcpusFlags(virDomainPtr dom, unsigned int nvcpus, ret = virDomainSaveConfig(driver->configDir, persistentDef); endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); + virDomainObjUnref(vm); return ret; } @@ -2645,7 +2650,7 @@ qemudDomainPinVcpu(virDomainPtr dom, virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } if (!virDomainObjIsActive(vm)) { @@ -2690,8 +2695,7 @@ qemudDomainPinVcpu(virDomainPtr dom, ret = 0; cleanup: - if (vm) - virDomainObjUnlock(vm); + virDomainObjUnref(vm); return ret; } @@ -2717,7 +2721,7 @@ qemudDomainGetVcpus(virDomainPtr dom, virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } if (!virDomainObjIsActive(vm)) { @@ -2780,8 +2784,7 @@ qemudDomainGetVcpus(virDomainPtr dom, ret = maxinfo; cleanup: - if (vm) - virDomainObjUnlock(vm); + virDomainObjUnref(vm); return ret; } @@ -3164,9 +3167,8 @@ qemuDomainRestore(virConnectPtr conn, ret = qemuDomainSaveImageStartVM(conn, driver, vm, &fd, &header, path); - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; - else if (ret < 0 && !vm->persistent) { + qemuDomainObjEndJob(vm); + if (ret < 0 && !vm->persistent) { virDomainRemoveInactive(&driver->domains, vm); vm = NULL; } @@ -3174,8 +3176,6 @@ qemuDomainRestore(virConnectPtr conn, cleanup: virDomainDefFree(def); VIR_FORCE_CLOSE(fd); - if (vm) - virDomainObjUnlock(vm); qemuDriverUnlock(driver); return ret; } @@ -3254,10 +3254,7 @@ static char *qemudDomainDumpXML(virDomainPtr dom, qemuDomainObjEnterMonitorWithDriver(driver, vm); err = qemuMonitorGetBalloonInfo(priv->mon, &balloon); qemuDomainObjExitMonitorWithDriver(driver, vm); - if (qemuDomainObjEndJob(vm) == 0) { - vm = NULL; - goto cleanup; - } + qemuDomainObjEndJob(vm); if (err < 0) goto cleanup; if (err > 0) @@ -3269,8 +3266,6 @@ static char *qemudDomainDumpXML(virDomainPtr dom, ret = qemuDomainFormatXML(driver, vm, flags); cleanup: - if (vm) - virDomainObjUnlock(vm); qemuDriverUnlock(driver); return ret; } @@ -3487,12 +3482,9 @@ qemudDomainStartWithFlags(virDomainPtr dom, unsigned int flags) (flags & VIR_DOMAIN_START_PAUSED) != 0); endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); qemuDriverUnlock(driver); return ret; } @@ -3859,8 +3851,7 @@ static int qemudDomainAttachDevice(virDomainPtr dom, ret = -1; endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: if (cgroup) @@ -3868,8 +3859,6 @@ cleanup: qemuCapsFree(qemuCaps); virDomainDeviceDefFree(dev); - if (vm) - virDomainObjUnlock(vm); qemuDriverUnlock(driver); return ret; } @@ -3993,8 +3982,7 @@ static int qemuDomainUpdateDeviceFlags(virDomainPtr dom, ret = -1; endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: if (cgroup) @@ -4002,8 +3990,6 @@ cleanup: qemuCapsFree(qemuCaps); virDomainDeviceDefFree(dev); - if (vm) - virDomainObjUnlock(vm); qemuDriverUnlock(driver); return ret; } @@ -4083,14 +4069,11 @@ static int qemudDomainDetachDevice(virDomainPtr dom, ret = -1; endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: qemuCapsFree(qemuCaps); virDomainDeviceDefFree(dev); - if (vm) - virDomainObjUnlock(vm); qemuDriverUnlock(driver); return ret; } @@ -4802,12 +4785,9 @@ qemudDomainBlockStats (virDomainPtr dom, qemuDomainObjExitMonitor(vm); endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); return ret; } @@ -4906,12 +4886,9 @@ qemudDomainMemoryStats (virDomainPtr dom, "%s", _("domain is not running")); } - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); return ret; } @@ -5064,15 +5041,12 @@ qemudDomainMemoryPeek (virDomainPtr dom, ret = 0; endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: VIR_FORCE_CLOSE(fd); unlink (tmp); VIR_FREE(tmp); - if (vm) - virDomainObjUnlock(vm); return ret; } @@ -5218,16 +5192,13 @@ static int qemuDomainGetBlockInfo(virDomainPtr dom, qemuDomainObjExitMonitor(vm); } - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); } else { ret = 0; } cleanup: VIR_FORCE_CLOSE(fd); - if (vm) - virDomainObjUnlock(vm); return ret; } @@ -6067,8 +6038,8 @@ cleanup: _("resuming after snapshot failed")); } - if (qemuDomainObjEndJob(vm) == 0) - *vmptr = NULL; + qemuDomainObjEndJob(vm); + *vmptr = NULL; return ret; } @@ -6438,9 +6409,8 @@ static int qemuDomainRevertToSnapshot(virDomainSnapshotPtr snapshot, VIR_DOMAIN_EVENT_STOPPED, VIR_DOMAIN_EVENT_STOPPED_FROM_SNAPSHOT); if (!vm->persistent) { - if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, vm); - vm = NULL; + qemuDomainObjEndJob(vm); + virDomainRemoveInactive(&driver->domains, vm); goto cleanup; } } @@ -6454,14 +6424,12 @@ static int qemuDomainRevertToSnapshot(virDomainSnapshotPtr snapshot, ret = 0; endjob: - if (vm && qemuDomainObjEndJob(vm) == 0) - vm = NULL; + if (vm) + qemuDomainObjEndJob(vm); cleanup: if (event) qemuDomainEventQueue(driver, event); - if (vm) - virDomainObjUnlock(vm); qemuDriverUnlock(driver); return ret; @@ -6675,12 +6643,9 @@ static int qemuDomainSnapshotDelete(virDomainSnapshotPtr snapshot, ret = qemuDomainSnapshotDiscard(driver, vm, snap); endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); qemuDriverUnlock(driver); return ret; } @@ -6727,14 +6692,9 @@ static int qemuDomainMonitorCommand(virDomainPtr domain, const char *cmd, qemuDomainObjEnterMonitorWithDriver(driver, vm); ret = qemuMonitorArbitraryCommand(priv->mon, cmd, result, hmp); qemuDomainObjExitMonitorWithDriver(driver, vm); - if (qemuDomainObjEndJob(vm) == 0) { - vm = NULL; - goto cleanup; - } + qemuDomainObjEndJob(vm); cleanup: - if (vm) - virDomainObjUnlock(vm); qemuDriverUnlock(driver); return ret; } diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 43741e1..462e6be 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -336,8 +336,8 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver, qemuAuditDomainStart(vm, "migrated", false); qemuProcessStop(driver, vm, 0); if (!vm->persistent) { - if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, vm); + qemuDomainObjEndJob(vm); + virDomainRemoveInactive(&driver->domains, vm); vm = NULL; } virReportSystemError(errno, "%s", @@ -353,9 +353,10 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver, ret = 0; endjob: - if (vm && - qemuDomainObjEndJob(vm) == 0) + if (vm) { + qemuDomainObjEndJob(vm); vm = NULL; + } /* We set a fake job active which is held across * API calls until the finish() call. This prevents @@ -374,11 +375,8 @@ cleanup: virDomainDefFree(def); VIR_FORCE_CLOSE(dataFD[0]); VIR_FORCE_CLOSE(dataFD[1]); - if (vm) - virDomainObjUnlock(vm); if (event) qemuDomainEventQueue(driver, event); - qemuDriverUnlock(driver); return ret; } @@ -530,8 +528,8 @@ qemuMigrationPrepareDirect(struct qemud_driver *driver, * should have already done that. */ if (!vm->persistent) { - if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, vm); + qemuDomainObjEndJob(vm); + virDomainRemoveInactive(&driver->domains, vm); vm = NULL; } goto endjob; @@ -544,9 +542,10 @@ qemuMigrationPrepareDirect(struct qemud_driver *driver, ret = 0; endjob: - if (vm && - qemuDomainObjEndJob(vm) == 0) + if (vm) { + qemuDomainObjEndJob(vm); vm = NULL; + } /* We set a fake job active which is held across * API calls until the finish() call. This prevents @@ -565,8 +564,6 @@ cleanup: virDomainDefFree(def); if (ret != 0) VIR_FREE(*uri_out); - if (vm) - virDomainObjUnlock(vm); if (event) qemuDomainEventQueue(driver, event); return ret; @@ -1090,8 +1087,8 @@ int qemuMigrationPerform(struct qemud_driver *driver, VIR_DOMAIN_EVENT_STOPPED_MIGRATED); if (!vm->persistent || (flags & VIR_MIGRATE_UNDEFINE_SOURCE)) { virDomainDeleteConfig(driver->configDir, driver->autostartDir, vm); - if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, vm); + qemuDomainObjEndJob(vm); + virDomainRemoveInactive(&driver->domains, vm); vm = NULL; } ret = 0; @@ -1112,13 +1109,12 @@ endjob: VIR_DOMAIN_EVENT_RESUMED, VIR_DOMAIN_EVENT_RESUMED_MIGRATED); } - if (vm && - qemuDomainObjEndJob(vm) == 0) + if (vm) { + qemuDomainObjEndJob(vm); vm = NULL; + } cleanup: - if (vm) - virDomainObjUnlock(vm); if (event) qemuDomainEventQueue(driver, event); return ret; @@ -1266,20 +1262,19 @@ qemuMigrationFinish(struct qemud_driver *driver, VIR_DOMAIN_EVENT_STOPPED, VIR_DOMAIN_EVENT_STOPPED_FAILED); if (!vm->persistent) { - if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, vm); + qemuDomainObjEndJob(vm); + virDomainRemoveInactive(&driver->domains, vm); vm = NULL; } } endjob: - if (vm && - qemuDomainObjEndJob(vm) == 0) + if (vm) { + qemuDomainObjEndJob(vm); vm = NULL; + } cleanup: - if (vm) - virDomainObjUnlock(vm); if (event) qemuDomainEventQueue(driver, event); return dom; diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 48ecd5c..244b22a 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -604,8 +604,8 @@ static void qemuProcessHandleMonitorDestroy(qemuMonitorPtr mon, priv = vm->privateData; if (priv->mon == mon) priv->mon = NULL; - if (virDomainObjUnref(vm) > 0) - virDomainObjUnlock(vm); + virDomainObjUnlock(vm); + virDomainObjUnref(vm); } static qemuMonitorCallbacks monitorCallbacks = { @@ -644,7 +644,7 @@ qemuConnectMonitor(struct qemud_driver *driver, virDomainObjPtr vm) /* Safe to ignore value since ref count was incremented above */ if (priv->mon == NULL) - ignore_value(virDomainObjUnref(vm)); + virDomainObjUnref(vm); if (virSecurityManagerClearSocketLabel(driver->securityManager, vm) < 0) { VIR_ERROR(_("Failed to clear security context for monitor for %s"), @@ -1940,10 +1940,6 @@ qemuProcessReconnect(void *payload, const void *name ATTRIBUTE_UNUSED, void *opa priv = obj->privateData; - /* Hold an extra reference because we can't allow 'vm' to be - * deleted if qemuConnectMonitor() failed */ - virDomainObjRef(obj); - /* XXX check PID liveliness & EXE path */ if (qemuConnectMonitor(driver, obj) < 0) goto error; @@ -1975,8 +1971,7 @@ qemuProcessReconnect(void *payload, const void *name ATTRIBUTE_UNUSED, void *opa if (obj->def->id >= driver->nextvmid) driver->nextvmid = obj->def->id + 1; - if (virDomainObjUnref(obj) > 0) - virDomainObjUnlock(obj); + virDomainObjUnlock(obj); qemuCapsFree(qemuCaps); return; @@ -1984,21 +1979,17 @@ qemuProcessReconnect(void *payload, const void *name ATTRIBUTE_UNUSED, void *opa error: qemuCapsFree(qemuCaps); if (!virDomainObjIsActive(obj)) { - if (virDomainObjUnref(obj) > 0) - virDomainObjUnlock(obj); + virDomainObjUnlock(obj); return; } - if (virDomainObjUnref(obj) > 0) { - /* We can't get the monitor back, so must kill the VM - * to remove danger of it ending up running twice if - * user tries to start it again later */ - qemuProcessStop(driver, obj, 0); - if (!obj->persistent) - virDomainRemoveInactive(&driver->domains, obj); - else - virDomainObjUnlock(obj); - } + /* We can't get the monitor back, so must kill the VM + * to remove danger of it ending up running twice if + * user tries to start it again later */ + qemuProcessStop(driver, obj, 0); + virDomainObjUnlock(obj); + if (!obj->persistent) + virDomainRemoveInactive(&driver->domains, obj); } /** diff --git a/src/vmware/vmware_conf.c b/src/vmware/vmware_conf.c index 6339248..6b59b18 100644 --- a/src/vmware/vmware_conf.c +++ b/src/vmware/vmware_conf.c @@ -205,7 +205,7 @@ cleanup: VIR_FREE(vmx); /* any non-NULL vm here has not been shared, so unref will return 0 */ if (vm) - ignore_value(virDomainObjUnref(vm)); + virDomainObjUnref(vm); return ret; } -- 1.7.3.1 -- Thanks, Hu Tao

On 04/06/2011 01:19 AM, Hu Tao wrote:
This patch also eliminates a dead-lock bug in qemuDomainObjBeginJobWithDriver: if virCondWaitUntil() timeouts, the thread tries to acquire qemu driver lock while holding virDomainObj lock.
Let's please separate that bug fix into a separate patch which gets applied as soon as possible.
--- src/conf/domain_conf.c | 56 ++++---- src/conf/domain_conf.h | 6 +- src/openvz/openvz_conf.c | 8 +- src/qemu/qemu_domain.c | 32 ++--- src/qemu/qemu_domain.h | 2 +- src/qemu/qemu_driver.c | 304 ++++++++++++++++++++------------------------- src/qemu/qemu_migration.c | 45 +++---- src/qemu/qemu_process.c | 33 ++--- src/vmware/vmware_conf.c | 2 +- 9 files changed, 215 insertions(+), 273 deletions(-)
int virDomainObjListInit(virDomainObjListPtr doms) @@ -437,7 +436,7 @@ virDomainObjPtr virDomainFindByID(const virDomainObjListPtr doms, virDomainObjPtr obj; obj = virHashSearch(doms->objs, virDomainObjListSearchID, &id); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj);
Wow - changing the semantics so the object is not locked by default, just referenced.
@@ -1010,6 +1007,11 @@ static virDomainObjPtr virDomainObjNew(virCapsPtr caps) return NULL; }
+ if (virObjectInit(&domain->obj, doDomainObjFree)) { + VIR_FREE(domain); + return NULL;
Hmm. virObjectInit used VIR_ERROR, which logs, but doesn't call into virtError. By returning NULL here, a user will get the dreaded "Unknown error" message since we didn't hook into the virterror machinery. Should virObjectInit instead be using virReportErrorHelper? Or is it considered a coding bug to ever have virObjectInit fail in the first place, so we don't really have to worry about that.
@@ -1075,8 +1075,10 @@ virDomainObjPtr virDomainAssignDef(virCapsPtr caps, domain->def = def;
virUUIDFormat(def->uuid, uuidstr); + virDomainObjRef(domain); if (virHashAddEntry(doms->objs, uuidstr, domain) < 0) {
It seems like incrementing ref count on container addition and decrementing it on removal are common actions. Would it simplify code any by making a wrapper for virHashAddEntry: int virHashAddObjEntry(virHashTablePtr, const void *name, virObjectPtr *data) which does the referencing as part of adding/removing a virObject from a hash table, rather than making all callers track it? The only potential drawback to that is that if you use the wrong function, the referencing doesn't happen. Or maybe even make virHashCreateFull take a bool parameter of whether the data must be a virObjectPtr, so you don't have to wrap virHashAddObjEntry (and since virHashRemoveEntry doesn't really have any way to create a virHashRemoveObjEntry wrapper, but it would need to be in on the game of automatic reference count manipulations any time we know the table hashes only virObjects as data).
@@ -1149,9 +1151,7 @@ virDomainObjGetPersistentDef(virCapsPtr caps, }
/* - * The caller must hold a lock on the driver owning 'doms', - * and must also have locked 'dom', to ensure no one else - * is either waiting for 'dom' or still usingn it + * The caller must hold a lock on the driver owning 'doms'.
While touching that line, fix the spacing: s/lock on/lock on/
@@ -1159,9 +1159,8 @@ void virDomainRemoveInactive(virDomainObjListPtr doms, char uuidstr[VIR_UUID_STRING_BUFLEN]; virUUIDFormat(dom->def->uuid, uuidstr);
- virDomainObjUnlock(dom); - virHashRemoveEntry(doms->objs, uuidstr); + virDomainObjUnref(dom);
Hmm, this means you are dereferencing dom->def->uuid while holding the driver lock but without holding the domain lock. If there is another place in code that holds the domain lock but not the driver lock, couldn't that cause a bad read of dom->def? I don't think you can blindly get rid of holding the lock on dom unless we make additional rules about which members of dom are safe to modify (for example, stating that dom->def cannot be modified unless you hold the driver lock), but that's hard to audit. :(
@@ -6146,7 +6145,7 @@ static virDomainObjPtr virDomainObjParseXML(virCapsPtr caps,
error: /* obj was never shared, so unref should return 0 */ - ignore_value(virDomainObjUnref(obj)); + virDomainObjUnref(obj);
Do we also want to delete the comment, since it only served to justify why we were using ignore_value()?
@@ -8449,10 +8448,12 @@ static virDomainObjPtr virDomainLoadConfig(virCapsPtr caps, if (!(dom = virDomainAssignDef(caps, doms, def, false))) goto error;
+ virDomainObjLock(dom); dom->autostart = autostart;
if (notify) (*notify)(dom, newVM, opaque); + virDomainObjUnlock(dom);
VIR_FREE(configFile); VIR_FREE(autostartLink);
Ouch, this changes virDomainLoadConfig so that it returns an unlocked domain, but virDomainLoadAllConfigs assumed that domain was still locked and you now have a double-unlock.
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..3a3c953 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -457,13 +457,13 @@ int qemuDomainObjBeginJob(virDomainObjPtr obj) } then = timeval_to_ms(now) + QEMU_JOB_WAIT_TIME;
- virDomainObjRef(obj); + virDomainObjLock(obj);
This grabbed obj->privateData while obj was not locked. Is that safe, or do you need to float the initialization of priv down?
while (priv->jobActive) { if (virCondWaitUntil(&priv->jobCond, &obj->lock, then) < 0) { - /* Safe to ignore value since ref count was incremented above */ - ignore_value(virDomainObjUnref(obj)); - if (errno == ETIMEDOUT) + int err = errno; + virDomainObjUnlock(obj); + if (err == ETIMEDOUT)
Good catch that pre-patch, errno was untouched by virDomainObjUnref, but post-patch, you need to preserve errno before unlocking.
qemuReportError(VIR_ERR_OPERATION_TIMEOUT, "%s", _("cannot acquire state change lock")); else @@ -482,12 +482,10 @@ int qemuDomainObjBeginJob(virDomainObjPtr obj) }
/* - * obj must be locked before calling, qemud_driver must be locked - * * This must be called by anything that will change the VM state * in any way, or anything that will use the QEMU monitor. */ -int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, +int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver ATTRIBUTE_UNUSED,
If we no longer care if driver was locked or unlocked, can we do a followup patch that simplifies all callers to just use qemuDomainObjBeginJob and delete this variant? But I don't think you can get away with it; see below...
virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; @@ -501,20 +499,18 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, } then = timeval_to_ms(now) + QEMU_JOB_WAIT_TIME;
- virDomainObjRef(obj); - qemuDriverUnlock(driver);
...don't think you can delete this line...
+ virDomainObjLock(obj);
Again, locking after you already grabbed priv.
while (priv->jobActive) { if (virCondWaitUntil(&priv->jobCond, &obj->lock, then) < 0) {
Ouch. You just broke the no sleeping while holding locks rule - previously, virCondWaitUntil was called with no locks, now you are calling it while holding both driver and domain lock. :(
- /* Safe to ignore value since ref count was incremented above */ - ignore_value(virDomainObjUnref(obj)); - if (errno == ETIMEDOUT) + int err = errno; + virDomainObjUnlock(obj); + if (err == ETIMEDOUT) qemuReportError(VIR_ERR_OPERATION_TIMEOUT, "%s", _("cannot acquire state change lock")); else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); - qemuDriverLock(driver);
Yep, this is the deadlock fix; pre-patch was definitely trying to regrab the driver lock while still holding the domain lock. But given that you _do_ have to regain driver lock (thanks to the blocking of virCondWaitUntil), I'm afraid your fix will instead have to look like...
return -1; } } @@ -524,10 +520,6 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, priv->jobStart = timeval_to_ms(now); memset(&priv->jobInfo, 0, sizeof(priv->jobInfo));
- virDomainObjUnlock(obj); - qemuDriverLock(driver); - virDomainObjLock(obj);
...these lines, which also must remain (although possibly modified a bit, if you are changing semantics to guarantee that you don't have to have obj locked prior to calling this function).
+++ b/src/qemu/qemu_driver.c @@ -139,7 +139,6 @@ qemuAutostartDomain(void *payload, const void *name ATTRIBUTE_UNUSED, void *opaq struct qemuAutostartData *data = opaque; virErrorPtr err;
- virDomainObjLock(vm); virResetLastError(); if (qemuDomainObjBeginJobWithDriver(data->driver, vm) < 0) { err = virGetLastError(); @@ -156,12 +155,8 @@ qemuAutostartDomain(void *payload, const void *name ATTRIBUTE_UNUSED, void *opaq err ? err->message : _("unknown error")); }
- if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); } - - if (vm) - virDomainObjUnlock(vm); }
Ah, so it qemuDomainObjBeginJobWithDriver is called while domain lock not held on entry but exits with it held on success, and qemuDomainObjEndJob is the counterpart that releases the domain lock. Makes for some more compact code everywhere else. :)
@@ -1239,39 +1243,57 @@ static virDomainPtr qemudDomainCreate(virConnectPtr conn, const char *xml,
qemuDriverLock(driver); if (!(def = virDomainDefParseString(driver->caps, xml, - VIR_DOMAIN_XML_INACTIVE))) + VIR_DOMAIN_XML_INACTIVE))) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (virSecurityManagerVerify(driver->securityManager, def) < 0) + if (virSecurityManagerVerify(driver->securityManager, def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (virDomainObjIsDuplicate(&driver->domains, def, 1) < 0) + if (virDomainObjIsDuplicate(&driver->domains, def, 1) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
Why must every caller unlock the driver, instead of factoring it once into the cleanup? I'd almost rather see a second label, as in: goto cleanup_locked; cleanup_locked: qemuDriverUnlock(driver); goto cleanup;
+ } + + qemuDriverUnlock(driver);
def = NULL;
- if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) {
With old semantics, you can't call qemuDomainObjBeginJobWithDriver if driver is unlocked (then again, you just changed the body of that method to not care if driver was locked, which matches with you just unlocking it a couple lines ago). Then again, maybe we don't need BeginJobWithDriver after all, if all of your conversions have changed things to no longer hold the driver lock before beginning a job.
@@ -1307,13 +1325,14 @@ static int qemudDomainSuspend(virDomainPtr dom) {
qemuDriverLock(driver); vm = virDomainFindByUUID(&driver->domains, dom->uuid); + qemuDriverUnlock(driver);
if (!vm) { char uuidstr[VIR_UUID_STRING_BUFLEN]; virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } if (!virDomainObjIsActive(vm)) { qemuReportError(VIR_ERR_OPERATION_INVALID, @@ -1354,16 +1373,12 @@ static int qemudDomainSuspend(virDomainPtr dom) { }
endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm);
Ouch - this change means you are accessing virDomainSaveStatus(driver->caps, driver->stateDir, vm) without holding the driver lock. Is that safe? (I don't know that caps or stateDir ever change after creation, so maybe it is safe after all). Likewise you call qemuProcessStopCPUs(driver, vm), which used to assume driver was locked - is that safe? I think a lot of this file will have the same questions, so I won't review the rest of qemu_domain.c very closely on this round.
@@ -1090,8 +1087,8 @@ int qemuMigrationPerform(struct qemud_driver *driver, VIR_DOMAIN_EVENT_STOPPED_MIGRATED); if (!vm->persistent || (flags & VIR_MIGRATE_UNDEFINE_SOURCE)) { virDomainDeleteConfig(driver->configDir, driver->autostartDir, vm); - if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, vm); + qemuDomainObjEndJob(vm); + virDomainRemoveInactive(&driver->domains, vm); vm = NULL;
Wonky spacing.
@@ -644,7 +644,7 @@ qemuConnectMonitor(struct qemud_driver *driver, virDomainObjPtr vm)
/* Safe to ignore value since ref count was incremented above */ if (priv->mon == NULL) - ignore_value(virDomainObjUnref(vm)); + virDomainObjUnref(vm);
Another case where the comment is out of date once you delete the ignore_value().
+++ b/src/vmware/vmware_conf.c @@ -205,7 +205,7 @@ cleanup: VIR_FREE(vmx); /* any non-NULL vm here has not been shared, so unref will return 0 */ if (vm) - ignore_value(virDomainObjUnref(vm)); + virDomainObjUnref(vm);
And another stale comment. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

On Wed, Apr 06, 2011 at 04:33:56PM -0600, Eric Blake wrote:
On 04/06/2011 01:19 AM, Hu Tao wrote:
This patch also eliminates a dead-lock bug in qemuDomainObjBeginJobWithDriver: if virCondWaitUntil() timeouts, the thread tries to acquire qemu driver lock while holding virDomainObj lock.
Let's please separate that bug fix into a separate patch which gets applied as soon as possible.
--- src/conf/domain_conf.c | 56 ++++---- src/conf/domain_conf.h | 6 +- src/openvz/openvz_conf.c | 8 +- src/qemu/qemu_domain.c | 32 ++--- src/qemu/qemu_domain.h | 2 +- src/qemu/qemu_driver.c | 304 ++++++++++++++++++++------------------------- src/qemu/qemu_migration.c | 45 +++---- src/qemu/qemu_process.c | 33 ++--- src/vmware/vmware_conf.c | 2 +- 9 files changed, 215 insertions(+), 273 deletions(-)
int virDomainObjListInit(virDomainObjListPtr doms) @@ -437,7 +436,7 @@ virDomainObjPtr virDomainFindByID(const virDomainObjListPtr doms, virDomainObjPtr obj; obj = virHashSearch(doms->objs, virDomainObjListSearchID, &id); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj);
Wow - changing the semantics so the object is not locked by default, just referenced.
@@ -1010,6 +1007,11 @@ static virDomainObjPtr virDomainObjNew(virCapsPtr caps) return NULL; }
+ if (virObjectInit(&domain->obj, doDomainObjFree)) { + VIR_FREE(domain); + return NULL;
Hmm. virObjectInit used VIR_ERROR, which logs, but doesn't call into virtError. By returning NULL here, a user will get the dreaded "Unknown error" message since we didn't hook into the virterror machinery. Should virObjectInit instead be using virReportErrorHelper? Or is it
How to use virReportErrorHelper in this case?
considered a coding bug to ever have virObjectInit fail in the first place, so we don't really have to worry about that.
Sorry for my bad English, I don't understand this sentence.
@@ -1075,8 +1075,10 @@ virDomainObjPtr virDomainAssignDef(virCapsPtr caps, domain->def = def;
virUUIDFormat(def->uuid, uuidstr); + virDomainObjRef(domain); if (virHashAddEntry(doms->objs, uuidstr, domain) < 0) {
It seems like incrementing ref count on container addition and decrementing it on removal are common actions. Would it simplify code any by making a wrapper for virHashAddEntry:
int virHashAddObjEntry(virHashTablePtr, const void *name, virObjectPtr *data)
which does the referencing as part of adding/removing a virObject from a hash table, rather than making all callers track it?
Agreed. But this needs many changes I think it is better to do it in a seperate patch.
The only potential drawback to that is that if you use the wrong function, the referencing doesn't happen. Or maybe even make virHashCreateFull take a bool parameter of whether the data must be a virObjectPtr, so you don't have to wrap virHashAddObjEntry (and since virHashRemoveEntry doesn't really have any way to create a virHashRemoveObjEntry wrapper, but it would need to be in on the game of automatic reference count manipulations any time we know the table hashes only virObjects as data).
Would it be better to have two types of hashtable, one hashes only virObjects, the other hashes data except virObjects? this can minimize the impact brought by the change of hashtable to existing code.
@@ -1149,9 +1151,7 @@ virDomainObjGetPersistentDef(virCapsPtr caps, }
/* - * The caller must hold a lock on the driver owning 'doms', - * and must also have locked 'dom', to ensure no one else - * is either waiting for 'dom' or still usingn it + * The caller must hold a lock on the driver owning 'doms'.
While touching that line, fix the spacing: s/lock on/lock on/
OK.
@@ -1159,9 +1159,8 @@ void virDomainRemoveInactive(virDomainObjListPtr doms, char uuidstr[VIR_UUID_STRING_BUFLEN]; virUUIDFormat(dom->def->uuid, uuidstr);
- virDomainObjUnlock(dom); - virHashRemoveEntry(doms->objs, uuidstr); + virDomainObjUnref(dom);
Hmm, this means you are dereferencing dom->def->uuid while holding the driver lock but without holding the domain lock. If there is another place in code that holds the domain lock but not the driver lock, couldn't that cause a bad read of dom->def? I don't think you can blindly get rid of holding the lock on dom unless we make additional rules about which members of dom are safe to modify (for example, stating that dom->def cannot be modified unless you hold the driver lock), but that's hard to audit. :(
Don't understand you very well here. Let me explain what I understand: If a dom is added into a hashtable, then it is not safe to modify its uuid since it is the key of dom in hashtable. And we have to make some rules about when the uuid can be modified(say, hold the driver lock first when dom is in hashtable). If I understand correctly, then I think the simple way to modify the uuid is to remove the dom from hashtable, modify uuid, then reinsert it into hashtable. The qemu driver lock here is to protect the hashtable objs(remove entry), not for unref dom. However, I think this code should be better like this: qemu driver lock dom = virHashRemoveEntry(); /* not be exactly but a way to get the removed dom */ qemu driver unlock unref dom
@@ -6146,7 +6145,7 @@ static virDomainObjPtr virDomainObjParseXML(virCapsPtr caps,
error: /* obj was never shared, so unref should return 0 */ - ignore_value(virDomainObjUnref(obj)); + virDomainObjUnref(obj);
Do we also want to delete the comment, since it only served to justify why we were using ignore_value()?
OK.
@@ -8449,10 +8448,12 @@ static virDomainObjPtr virDomainLoadConfig(virCapsPtr caps, if (!(dom = virDomainAssignDef(caps, doms, def, false))) goto error;
+ virDomainObjLock(dom); dom->autostart = autostart;
if (notify) (*notify)(dom, newVM, opaque); + virDomainObjUnlock(dom);
VIR_FREE(configFile); VIR_FREE(autostartLink);
Ouch, this changes virDomainLoadConfig so that it returns an unlocked domain, but virDomainLoadAllConfigs assumed that domain was still locked and you now have a double-unlock.
Will check this carefully.
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..3a3c953 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -457,13 +457,13 @@ int qemuDomainObjBeginJob(virDomainObjPtr obj) } then = timeval_to_ms(now) + QEMU_JOB_WAIT_TIME;
- virDomainObjRef(obj); + virDomainObjLock(obj);
This grabbed obj->privateData while obj was not locked. Is that safe, or do you need to float the initialization of priv down?
Yes, you are right.
while (priv->jobActive) { if (virCondWaitUntil(&priv->jobCond, &obj->lock, then) < 0) { - /* Safe to ignore value since ref count was incremented above */ - ignore_value(virDomainObjUnref(obj)); - if (errno == ETIMEDOUT) + int err = errno; + virDomainObjUnlock(obj); + if (err == ETIMEDOUT)
Good catch that pre-patch, errno was untouched by virDomainObjUnref, but post-patch, you need to preserve errno before unlocking.
qemuReportError(VIR_ERR_OPERATION_TIMEOUT, "%s", _("cannot acquire state change lock")); else @@ -482,12 +482,10 @@ int qemuDomainObjBeginJob(virDomainObjPtr obj) }
/* - * obj must be locked before calling, qemud_driver must be locked - * * This must be called by anything that will change the VM state * in any way, or anything that will use the QEMU monitor. */ -int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, +int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver ATTRIBUTE_UNUSED,
If we no longer care if driver was locked or unlocked, can we do a followup patch that simplifies all callers to just use qemuDomainObjBeginJob and delete this variant? But I don't think you can get away with it; see below...
virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; @@ -501,20 +499,18 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, } then = timeval_to_ms(now) + QEMU_JOB_WAIT_TIME;
- virDomainObjRef(obj); - qemuDriverUnlock(driver);
...don't think you can delete this line...
+ virDomainObjLock(obj);
Again, locking after you already grabbed priv.
while (priv->jobActive) { if (virCondWaitUntil(&priv->jobCond, &obj->lock, then) < 0) {
Ouch. You just broke the no sleeping while holding locks rule - previously, virCondWaitUntil was called with no locks, now you are calling it while holding both driver and domain lock. :(
- /* Safe to ignore value since ref count was incremented above */ - ignore_value(virDomainObjUnref(obj)); - if (errno == ETIMEDOUT) + int err = errno; + virDomainObjUnlock(obj); + if (err == ETIMEDOUT) qemuReportError(VIR_ERR_OPERATION_TIMEOUT, "%s", _("cannot acquire state change lock")); else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); - qemuDriverLock(driver);
Yep, this is the deadlock fix; pre-patch was definitely trying to regrab the driver lock while still holding the domain lock. But given that you _do_ have to regain driver lock (thanks to the blocking of virCondWaitUntil), I'm afraid your fix will instead have to look like...
return -1; } } @@ -524,10 +520,6 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, priv->jobStart = timeval_to_ms(now); memset(&priv->jobInfo, 0, sizeof(priv->jobInfo));
- virDomainObjUnlock(obj); - qemuDriverLock(driver); - virDomainObjLock(obj);
...these lines, which also must remain (although possibly modified a bit, if you are changing semantics to guarantee that you don't have to have obj locked prior to calling this function).
+++ b/src/qemu/qemu_driver.c @@ -139,7 +139,6 @@ qemuAutostartDomain(void *payload, const void *name ATTRIBUTE_UNUSED, void *opaq struct qemuAutostartData *data = opaque; virErrorPtr err;
- virDomainObjLock(vm); virResetLastError(); if (qemuDomainObjBeginJobWithDriver(data->driver, vm) < 0) { err = virGetLastError(); @@ -156,12 +155,8 @@ qemuAutostartDomain(void *payload, const void *name ATTRIBUTE_UNUSED, void *opaq err ? err->message : _("unknown error")); }
- if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); } - - if (vm) - virDomainObjUnlock(vm); }
Ah, so it qemuDomainObjBeginJobWithDriver is called while domain lock not held on entry but exits with it held on success, and qemuDomainObjEndJob is the counterpart that releases the domain lock. Makes for some more compact code everywhere else. :)
@@ -1239,39 +1243,57 @@ static virDomainPtr qemudDomainCreate(virConnectPtr conn, const char *xml,
qemuDriverLock(driver); if (!(def = virDomainDefParseString(driver->caps, xml, - VIR_DOMAIN_XML_INACTIVE))) + VIR_DOMAIN_XML_INACTIVE))) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (virSecurityManagerVerify(driver->securityManager, def) < 0) + if (virSecurityManagerVerify(driver->securityManager, def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (virDomainObjIsDuplicate(&driver->domains, def, 1) < 0) + if (virDomainObjIsDuplicate(&driver->domains, def, 1) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
Why must every caller unlock the driver, instead of factoring it once into the cleanup? I'd almost rather see a second label, as in:
goto cleanup_locked;
cleanup_locked: qemuDriverUnlock(driver); goto cleanup;
OK, makes code cleaner.
+ } + + qemuDriverUnlock(driver);
def = NULL;
- if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) {
With old semantics, you can't call qemuDomainObjBeginJobWithDriver if driver is unlocked (then again, you just changed the body of that method to not care if driver was locked, which matches with you just unlocking it a couple lines ago). Then again, maybe we don't need BeginJobWithDriver after all, if all of your conversions have changed things to no longer hold the driver lock before beginning a job.
@@ -1307,13 +1325,14 @@ static int qemudDomainSuspend(virDomainPtr dom) {
qemuDriverLock(driver); vm = virDomainFindByUUID(&driver->domains, dom->uuid); + qemuDriverUnlock(driver);
if (!vm) { char uuidstr[VIR_UUID_STRING_BUFLEN]; virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } if (!virDomainObjIsActive(vm)) { qemuReportError(VIR_ERR_OPERATION_INVALID, @@ -1354,16 +1373,12 @@ static int qemudDomainSuspend(virDomainPtr dom) { }
endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm);
Ouch - this change means you are accessing virDomainSaveStatus(driver->caps, driver->stateDir, vm) without holding the driver lock. Is that safe? (I don't know that caps or stateDir ever change after creation, so maybe it is safe after all). Likewise
Yes, not safe here. Since the semantics of qemuDomainObjBeginJobWithDriver is changed, and many places call it, I have to check every place and test the change. In next version I plan to split the series into patches like this: first a patch changes virDomainObj to inherit from virObject, second a patch changes qemuDomainObjBeginJobWithDriver, then each patch for a place that calls qemuDomainObjBeginJobWithDriver. Is this reasonable? Or a better way?
you call qemuProcessStopCPUs(driver, vm), which used to assume driver was locked - is that safe?
As you have already noticed, the semantics of qemuDomainObjBeginJobWithDriver and qemuEnterMonitorWithDriver(patch 4) are changed, qemu driver lock have not to be hold before calling them. So if this patch is applied, patch 4 should also be applied.
I think a lot of this file will have the same questions, so I won't review the rest of qemu_domain.c very closely on this round.
@@ -1090,8 +1087,8 @@ int qemuMigrationPerform(struct qemud_driver *driver, VIR_DOMAIN_EVENT_STOPPED_MIGRATED); if (!vm->persistent || (flags & VIR_MIGRATE_UNDEFINE_SOURCE)) { virDomainDeleteConfig(driver->configDir, driver->autostartDir, vm); - if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, vm); + qemuDomainObjEndJob(vm); + virDomainRemoveInactive(&driver->domains, vm); vm = NULL;
Wonky spacing.
@@ -644,7 +644,7 @@ qemuConnectMonitor(struct qemud_driver *driver, virDomainObjPtr vm)
/* Safe to ignore value since ref count was incremented above */ if (priv->mon == NULL) - ignore_value(virDomainObjUnref(vm)); + virDomainObjUnref(vm);
Another case where the comment is out of date once you delete the ignore_value().
Will remove it.
+++ b/src/vmware/vmware_conf.c @@ -205,7 +205,7 @@ cleanup: VIR_FREE(vmx); /* any non-NULL vm here has not been shared, so unref will return 0 */ if (vm) - ignore_value(virDomainObjUnref(vm)); + virDomainObjUnref(vm);
And another stale comment.
Will remove it.
-- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

On 04/07/2011 03:41 AM, Hu Tao wrote:
@@ -1010,6 +1007,11 @@ static virDomainObjPtr virDomainObjNew(virCapsPtr caps) return NULL; }
+ if (virObjectInit(&domain->obj, doDomainObjFree)) { + VIR_FREE(domain); + return NULL;
Hmm. virObjectInit used VIR_ERROR, which logs, but doesn't call into virtError. By returning NULL here, a user will get the dreaded "Unknown error" message since we didn't hook into the virterror machinery. Should virObjectInit instead be using virReportErrorHelper? Or is it
How to use virReportErrorHelper in this case?
considered a coding bug to ever have virObjectInit fail in the first place, so we don't really have to worry about that.
Sorry for my bad English, I don't understand this sentence.
Basically, I was envisioning: # define virObjectReportError(code, ...) \ virReportErrorHelper(NULL, VIR_FROM_NONE, code, __FILE__, \ __FUNCTION__, __LINE__, __VA_ARGS__) and calling virObjectReportError() instead of VIR_ERROR() for any failure inside of virobject.c. VIR_ERROR only hooks up to the log file, but virReportErrorHelper _also_ hooks up to the remote procedure call (which means that if the error happens due to someone making an API call, they get the message back rather than just a cryptic "Unknown error"). But if we can guarantee that the only possible errors are due to coding mistakes (and in reality, that's true), then I'd almost go so far as to do make virObjectInit return void instead of int (callers need not check for failure), and inside virObjectInit do: if (obj->free) { VIR_ERROR ("Coding bug encountered, aborting"); abort (); } Gnulib already has uses of abort() in places that can only be reached by pure coding bugs, even though libvirt.git does not currently have anything like that. On the other hand, there's a difference between errors in virObjectInit() (which are only possible due to severe coding bugs and easy to audit that they will never hit without ever risking an abort() escaping into released code) vs. errors in virObjectUnref() (if we've unreferenced an object too many times, it's hard to tell which place in the code was the culprit, so the bug may escape in to a release and a library should avoid abort() at all costs in released code). At any rate, any time we hit a reference counting bug, it is in our best interest to do something loud and obvious that a bug occurred, to make it easier to fix the bug, rather than trying to proceed with invalid data.
The only potential drawback to that is that if you use the wrong function, the referencing doesn't happen. Or maybe even make virHashCreateFull take a bool parameter of whether the data must be a virObjectPtr, so you don't have to wrap virHashAddObjEntry (and since virHashRemoveEntry doesn't really have any way to create a virHashRemoveObjEntry wrapper, but it would need to be in on the game of automatic reference count manipulations any time we know the table hashes only virObjects as data).
Would it be better to have two types of hashtable, one hashes only virObjects, the other hashes data except virObjects? this can minimize the impact brought by the change of hashtable to existing code.
Proving something is a virObject can be done via type-safe wrapper functions, but only for functions that take a data argument in the first place (virHashRemoveEntry does not). But in the opposite direction, how can you prove something is not a virObject? I don't think you can forbid that (nor do you necessarily want to; it might make sense to hash void* which happens to be virObject but where the container does not plan on owning the object). I think the best you can do is adding helper methods that operate only if you request and promise to only use virObject as the data, and adding a bool flag to virHashCreateFull seems the least invasive way to do it.
If a dom is added into a hashtable, then it is not safe to modify its uuid since it is the key of dom in hashtable. And we have to make some rules about when the uuid can be modified(say, hold the driver lock first when dom is in hashtable).
Yes, for a given object, especially if that object is hashed by a subset of itself, then the data being used as a hash key must not be modified. And we don't modify domain uuid's after creation (if we need a new uuid, we are creating a new domain rather than modifying an existing one). But without documenting which fields are constant and can be accessed without obtaining a lock and which must not be modified after the constructor, compared to the remaining fields that must only be read or written while holding the lock, is on a case-by-case basis per struct that inherits from virObject. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org


Sorry, I unexpectedly deleted text body. I changed the code like this: diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..8aad0b3 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -514,7 +514,10 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); + virDomainObjUnlock(obj); qemuDriverLock(driver); + virDomainObjLock(obj); + virDomainObjUnref(obj); return -1; } } but still have problem. (see log in previous mail.) The reference count of virDomainObj becomes 0 while there are still threads accessing it. This causes two results: one is segmentation fault as shown in the log; the other is the qemu monitor fd is closed two early, and threads are polling a closed fd. Steps of my test: 1. change the value of max_clients in /etc/libvirt/libvirtd.conf to big enough value: max_clients = 2000000 2. run libvirtd 3. run libvirt-test.sh hut-vm1 (hut-vm1 is my vm name) (warning: this script forks many processes you have to kill by hand) (again, the log and script libvirt-test.sh are in my previous mail)

At 04/12/2011 01:51 PM, Hu Tao Write:
Sorry, I unexpectedly deleted text body.
I changed the code like this:
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..8aad0b3 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -514,7 +514,10 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); + virDomainObjUnlock(obj); qemuDriverLock(driver); + virDomainObjLock(obj);
We lock qemu_driver and vm as the folling steps: 1. lock qemu_driver 2. lock vm We try to lock qemu_driver while holding the vm's lock. OOps, it will cause the libvirtd deadlock.(I can reproduce this bug by your script)
+ virDomainObjUnref(obj);
We have unref it, so do not unref it again.
return -1; } }
but still have problem. (see log in previous mail.) The reference count of virDomainObj becomes 0 while there are still threads accessing it. This causes two results: one is segmentation fault as shown in the log; the other is the qemu monitor fd is closed two early, and threads are polling a closed fd.
Steps of my test:
1. change the value of max_clients in /etc/libvirt/libvirtd.conf to big enough value:
max_clients = 2000000
2. run libvirtd
3. run libvirt-test.sh hut-vm1 (hut-vm1 is my vm name) (warning: this script forks many processes you have to kill by hand)
(again, the log and script libvirt-test.sh are in my previous mail)
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

On Tue, Apr 12, 2011 at 06:22:12PM +0800, Wen Congyang wrote:
At 04/12/2011 01:51 PM, Hu Tao Write:
Sorry, I unexpectedly deleted text body.
I changed the code like this:
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..8aad0b3 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -514,7 +514,10 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); + virDomainObjUnlock(obj); qemuDriverLock(driver); + virDomainObjLock(obj);
We lock qemu_driver and vm as the folling steps: 1. lock qemu_driver 2. lock vm
We try to lock qemu_driver while holding the vm's lock. OOps, it will cause the libvirtd deadlock.(I can reproduce this bug by your script)
+ virDomainObjUnref(obj);
We have unref it, so do not unref it again.
Didn't see there is already a call to virDomainObjUnref. the patch should be:
From 712883d0151222a276f777f473d96aa23ad5d9d6 Mon Sep 17 00:00:00 2001 From: Hu Tao <hutao@cn.fujitsu.com> Date: Tue, 12 Apr 2011 18:29:27 +0800 Subject: [PATCH] qemu: fix a dead-lock problem
In qemuDomainObjBeginJobWithDriver, when virCondWaitUntil timeouts, the function tries to call qemuDriverLock with virDomainObj locked, this causes the dead-lock problem. This patch fixes this. --- src/qemu/qemu_domain.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..a947b4e 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -506,15 +506,17 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, while (priv->jobActive) { if (virCondWaitUntil(&priv->jobCond, &obj->lock, then) < 0) { - /* Safe to ignore value since ref count was incremented above */ - ignore_value(virDomainObjUnref(obj)); if (errno == ETIMEDOUT) qemuReportError(VIR_ERR_OPERATION_TIMEOUT, "%s", _("cannot acquire state change lock")); else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); + virDomainObjUnlock(obj); qemuDriverLock(driver); + virDomainObjLock(obj); + /* Safe to ignore value since ref count was incremented above */ + ignore_value(virDomainObjUnref(obj)); return -1; } } -- 1.7.3.1

On Tue, Apr 12, 2011 at 06:33:21PM +0800, Hu Tao wrote:
On Tue, Apr 12, 2011 at 06:22:12PM +0800, Wen Congyang wrote:
At 04/12/2011 01:51 PM, Hu Tao Write:
Sorry, I unexpectedly deleted text body.
I changed the code like this:
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..8aad0b3 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -514,7 +514,10 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); + virDomainObjUnlock(obj); qemuDriverLock(driver); + virDomainObjLock(obj);
We lock qemu_driver and vm as the folling steps: 1. lock qemu_driver 2. lock vm
We try to lock qemu_driver while holding the vm's lock. OOps, it will cause the libvirtd deadlock.(I can reproduce this bug by your script)
+ virDomainObjUnref(obj);
We have unref it, so do not unref it again.
Didn't see there is already a call to virDomainObjUnref. the patch should be:
From 712883d0151222a276f777f473d96aa23ad5d9d6 Mon Sep 17 00:00:00 2001 From: Hu Tao <hutao@cn.fujitsu.com> Date: Tue, 12 Apr 2011 18:29:27 +0800 Subject: [PATCH] qemu: fix a dead-lock problem
In qemuDomainObjBeginJobWithDriver, when virCondWaitUntil timeouts, the function tries to call qemuDriverLock with virDomainObj locked, this causes the dead-lock problem. This patch fixes this. --- src/qemu/qemu_domain.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..a947b4e 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -506,15 +506,17 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver,
while (priv->jobActive) { if (virCondWaitUntil(&priv->jobCond, &obj->lock, then) < 0) { - /* Safe to ignore value since ref count was incremented above */ - ignore_value(virDomainObjUnref(obj)); if (errno == ETIMEDOUT) qemuReportError(VIR_ERR_OPERATION_TIMEOUT, "%s", _("cannot acquire state change lock")); else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); + virDomainObjUnlock(obj); qemuDriverLock(driver); + virDomainObjLock(obj); + /* Safe to ignore value since ref count was incremented above */ + ignore_value(virDomainObjUnref(obj)); return -1; } } --
ACK Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

At 04/12/2011 06:40 PM, Daniel P. Berrange Write:
On Tue, Apr 12, 2011 at 06:33:21PM +0800, Hu Tao wrote:
On Tue, Apr 12, 2011 at 06:22:12PM +0800, Wen Congyang wrote:
At 04/12/2011 01:51 PM, Hu Tao Write:
Sorry, I unexpectedly deleted text body.
I changed the code like this:
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..8aad0b3 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -514,7 +514,10 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); + virDomainObjUnlock(obj); qemuDriverLock(driver); + virDomainObjLock(obj);
We lock qemu_driver and vm as the folling steps: 1. lock qemu_driver 2. lock vm
We try to lock qemu_driver while holding the vm's lock. OOps, it will cause the libvirtd deadlock.(I can reproduce this bug by your script)
+ virDomainObjUnref(obj);
We have unref it, so do not unref it again.
Didn't see there is already a call to virDomainObjUnref. the patch should be:
From 712883d0151222a276f777f473d96aa23ad5d9d6 Mon Sep 17 00:00:00 2001 From: Hu Tao <hutao@cn.fujitsu.com> Date: Tue, 12 Apr 2011 18:29:27 +0800 Subject: [PATCH] qemu: fix a dead-lock problem
In qemuDomainObjBeginJobWithDriver, when virCondWaitUntil timeouts, the function tries to call qemuDriverLock with virDomainObj locked, this causes the dead-lock problem. This patch fixes this. --- src/qemu/qemu_domain.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index c2a1f9a..a947b4e 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -506,15 +506,17 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver,
while (priv->jobActive) { if (virCondWaitUntil(&priv->jobCond, &obj->lock, then) < 0) { - /* Safe to ignore value since ref count was incremented above */ - ignore_value(virDomainObjUnref(obj)); if (errno == ETIMEDOUT) qemuReportError(VIR_ERR_OPERATION_TIMEOUT, "%s", _("cannot acquire state change lock")); else virReportSystemError(errno, "%s", _("cannot acquire job mutex")); + virDomainObjUnlock(obj); qemuDriverLock(driver); + virDomainObjLock(obj); + /* Safe to ignore value since ref count was incremented above */ + ignore_value(virDomainObjUnref(obj)); return -1; } } --
ACK
Thanks. Applied.
Daniel

On Wed, Apr 06, 2011 at 03:19:51PM +0800, Hu Tao wrote:
This patch also eliminates a dead-lock bug in qemuDomainObjBeginJobWithDriver: if virCondWaitUntil() timeouts, the thread tries to acquire qemu driver lock while holding virDomainObj lock. --- src/conf/domain_conf.c | 56 ++++---- src/conf/domain_conf.h | 6 +- src/openvz/openvz_conf.c | 8 +- src/qemu/qemu_domain.c | 32 ++--- src/qemu/qemu_domain.h | 2 +- src/qemu/qemu_driver.c | 304 ++++++++++++++++++++------------------------- src/qemu/qemu_migration.c | 45 +++---- src/qemu/qemu_process.c | 33 ++--- src/vmware/vmware_conf.c | 2 +- 9 files changed, 215 insertions(+), 273 deletions(-)
diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index 90a1317..fc76a00 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -47,6 +47,7 @@ #include "storage_file.h" #include "files.h" #include "bitmap.h" +#include "virobject.h"
#define VIR_FROM_THIS VIR_FROM_DOMAIN
@@ -395,9 +396,7 @@ static void virDomainObjListDataFree(void *payload, const void *name ATTRIBUTE_UNUSED) { virDomainObjPtr obj = payload; - virDomainObjLock(obj); - if (virDomainObjUnref(obj) > 0) - virDomainObjUnlock(obj); + virDomainObjUnref(obj); }
int virDomainObjListInit(virDomainObjListPtr doms) @@ -437,7 +436,7 @@ virDomainObjPtr virDomainFindByID(const virDomainObjListPtr doms, virDomainObjPtr obj; obj = virHashSearch(doms->objs, virDomainObjListSearchID, &id); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj); return obj; }
@@ -452,7 +451,7 @@ virDomainObjPtr virDomainFindByUUID(const virDomainObjListPtr doms,
obj = virHashLookup(doms->objs, uuidstr); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj); return obj; }
@@ -476,7 +475,7 @@ virDomainObjPtr virDomainFindByName(const virDomainObjListPtr doms, virDomainObjPtr obj; obj = virHashSearch(doms->objs, virDomainObjListSearchName, name); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj); return obj; }
This is a major change in semantics, which makes pretty much every single caller non-thread safe, unless the callers are all changed todo virDomainObjLock immediately after calling this. So I don't really see the point in this - it just means more code duplication.
@@ -967,6 +966,12 @@ static void virDomainObjFree(virDomainObjPtr dom) { if (!dom) return; + virDomainObjUnref(dom); +} + +static void doDomainObjFree(virObjectPtr obj) +{ + virDomainObjPtr dom = (virDomainObjPtr)obj;
VIR_DEBUG("obj=%p", dom); virDomainDefFree(dom->def); @@ -984,21 +989,13 @@ static void virDomainObjFree(virDomainObjPtr dom)
void virDomainObjRef(virDomainObjPtr dom) { - dom->refs++; - VIR_DEBUG("obj=%p refs=%d", dom, dom->refs); + virObjectRef(&dom->obj); }
-int virDomainObjUnref(virDomainObjPtr dom) +void virDomainObjUnref(virDomainObjPtr dom) { - dom->refs--; - VIR_DEBUG("obj=%p refs=%d", dom, dom->refs); - if (dom->refs == 0) { - virDomainObjUnlock(dom); - virDomainObjFree(dom); - return 0; - } - return dom->refs; + virObjectUnref(&dom->obj); }
static virDomainObjPtr virDomainObjNew(virCapsPtr caps) @@ -1010,6 +1007,11 @@ static virDomainObjPtr virDomainObjNew(virCapsPtr caps) return NULL; }
+ if (virObjectInit(&domain->obj, doDomainObjFree)) { + VIR_FREE(domain); + return NULL; + } + if (caps->privateDataAllocFunc && !(domain->privateData = (caps->privateDataAllocFunc)())) { virReportOOMError(); @@ -1027,9 +1029,7 @@ static virDomainObjPtr virDomainObjNew(virCapsPtr caps) return NULL; }
- virDomainObjLock(domain); domain->state = VIR_DOMAIN_SHUTOFF; - domain->refs = 1;
virDomainSnapshotObjListInit(&domain->snapshots);
@@ -1075,8 +1075,10 @@ virDomainObjPtr virDomainAssignDef(virCapsPtr caps, domain->def = def;
virUUIDFormat(def->uuid, uuidstr); + virDomainObjRef(domain); if (virHashAddEntry(doms->objs, uuidstr, domain) < 0) { - VIR_FREE(domain); + virDomainObjUnref(domain); + virDomainObjFree(domain); return NULL; }
Simiarly here, you're now requiring all callers to manually obtain a lock.
@@ -1239,39 +1243,57 @@ static virDomainPtr qemudDomainCreate(virConnectPtr conn, const char *xml,
qemuDriverLock(driver); if (!(def = virDomainDefParseString(driver->caps, xml, - VIR_DOMAIN_XML_INACTIVE))) + VIR_DOMAIN_XML_INACTIVE))) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (virSecurityManagerVerify(driver->securityManager, def) < 0) + if (virSecurityManagerVerify(driver->securityManager, def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (virDomainObjIsDuplicate(&driver->domains, def, 1) < 0) + if (virDomainObjIsDuplicate(&driver->domains, def, 1) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (qemudCanonicalizeMachine(driver, def) < 0) + if (qemudCanonicalizeMachine(driver, def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (qemuDomainAssignPCIAddresses(def) < 0) + if (qemuDomainAssignPCIAddresses(def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
if (!(vm = virDomainAssignDef(driver->caps, &driver->domains, - def, false))) + def, false))) { + qemuDriverUnlock(driver); goto cleanup; + } + + qemuDriverUnlock(driver);
driver is now unlocked....
def = NULL;
- if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) { + virDomainObjUnref(vm);
...but this method *requires* driver to be locked.
goto cleanup; /* XXXX free the 'vm' we created ? */ + }
if (qemuProcessStart(conn, driver, vm, NULL, (flags & VIR_DOMAIN_START_PAUSED) != 0, -1, NULL, VIR_VM_OP_CREATE) < 0) { qemuAuditDomainStart(vm, "booted", false);
...and this method writes to 'driver', so it is now unsafe.
- if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, - vm); - vm = NULL; + qemuDomainObjEndJob(vm); + qemuDriverLock(driver); + virDomainRemoveInactive(&driver->domains, + vm); + qemuDriverUnlock(driver); + virDomainObjUnref(vm); goto cleanup; }
@@ -1283,17 +1305,13 @@ static virDomainPtr qemudDomainCreate(virConnectPtr conn, const char *xml, dom = virGetDomain(conn, vm->def->name, vm->def->uuid); if (dom) dom->id = vm->def->id;
- if (vm && - qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); + virDomainObjUnref(vm);
cleanup: virDomainDefFree(def); - if (vm) - virDomainObjUnlock(vm); if (event) qemuDomainEventQueue(driver, event); - qemuDriverUnlock(driver); return dom; }
And all the usage of 'vm' in this method is now completely unlocked and unsafe.
@@ -1307,13 +1325,14 @@ static int qemudDomainSuspend(virDomainPtr dom) {
qemuDriverLock(driver); vm = virDomainFindByUUID(&driver->domains, dom->uuid); + qemuDriverUnlock(driver);
if (!vm) { char uuidstr[VIR_UUID_STRING_BUFLEN]; virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } if (!virDomainObjIsActive(vm)) { qemuReportError(VIR_ERR_OPERATION_INVALID, @@ -1354,16 +1373,12 @@ static int qemudDomainSuspend(virDomainPtr dom) { }
endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm);
cleanup: - if (vm) - virDomainObjUnlock(vm); - + virDomainObjUnref(vm); if (event) qemuDomainEventQueue(driver, event); - qemuDriverUnlock(driver); return ret; }
Also now completely unsafe since 'vm' is never locked while making changes to it. [cut rest of patch] I don't see how any of this patch is threadsafe now that virtually no methods are acquiring the 'vm' lock. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Thu, Apr 07, 2011 at 10:33:03AM +0100, Daniel P. Berrange wrote:
On Wed, Apr 06, 2011 at 03:19:51PM +0800, Hu Tao wrote:
This patch also eliminates a dead-lock bug in qemuDomainObjBeginJobWithDriver: if virCondWaitUntil() timeouts, the thread tries to acquire qemu driver lock while holding virDomainObj lock. --- src/conf/domain_conf.c | 56 ++++---- src/conf/domain_conf.h | 6 +- src/openvz/openvz_conf.c | 8 +- src/qemu/qemu_domain.c | 32 ++--- src/qemu/qemu_domain.h | 2 +- src/qemu/qemu_driver.c | 304 ++++++++++++++++++++------------------------- src/qemu/qemu_migration.c | 45 +++---- src/qemu/qemu_process.c | 33 ++--- src/vmware/vmware_conf.c | 2 +- 9 files changed, 215 insertions(+), 273 deletions(-)
diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index 90a1317..fc76a00 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -47,6 +47,7 @@ #include "storage_file.h" #include "files.h" #include "bitmap.h" +#include "virobject.h"
#define VIR_FROM_THIS VIR_FROM_DOMAIN
@@ -395,9 +396,7 @@ static void virDomainObjListDataFree(void *payload, const void *name ATTRIBUTE_UNUSED) { virDomainObjPtr obj = payload; - virDomainObjLock(obj); - if (virDomainObjUnref(obj) > 0) - virDomainObjUnlock(obj); + virDomainObjUnref(obj); }
int virDomainObjListInit(virDomainObjListPtr doms) @@ -437,7 +436,7 @@ virDomainObjPtr virDomainFindByID(const virDomainObjListPtr doms, virDomainObjPtr obj; obj = virHashSearch(doms->objs, virDomainObjListSearchID, &id); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj); return obj; }
@@ -452,7 +451,7 @@ virDomainObjPtr virDomainFindByUUID(const virDomainObjListPtr doms,
obj = virHashLookup(doms->objs, uuidstr); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj); return obj; }
@@ -476,7 +475,7 @@ virDomainObjPtr virDomainFindByName(const virDomainObjListPtr doms, virDomainObjPtr obj; obj = virHashSearch(doms->objs, virDomainObjListSearchName, name); if (obj) - virDomainObjLock(obj); + virDomainObjRef(obj); return obj; }
This is a major change in semantics, which makes pretty much every single caller non-thread safe, unless the callers are all changed todo virDomainObjLock immediately after calling this. So I don't really see the point in this - it just means more code duplication.
If we do this: obj = virHashSearch(doms->objs, virDomainObjListSearchName, name); if (obj) virDomainObjLock(obj); return obj; And at the meantime another thread removes the same obj from doms->objs and frees it, than we are accessing a freed obj. lock doesn't help prevent object from being freed.
@@ -967,6 +966,12 @@ static void virDomainObjFree(virDomainObjPtr dom) { if (!dom) return; + virDomainObjUnref(dom); +} + +static void doDomainObjFree(virObjectPtr obj) +{ + virDomainObjPtr dom = (virDomainObjPtr)obj;
VIR_DEBUG("obj=%p", dom); virDomainDefFree(dom->def); @@ -984,21 +989,13 @@ static void virDomainObjFree(virDomainObjPtr dom)
void virDomainObjRef(virDomainObjPtr dom) { - dom->refs++; - VIR_DEBUG("obj=%p refs=%d", dom, dom->refs); + virObjectRef(&dom->obj); }
-int virDomainObjUnref(virDomainObjPtr dom) +void virDomainObjUnref(virDomainObjPtr dom) { - dom->refs--; - VIR_DEBUG("obj=%p refs=%d", dom, dom->refs); - if (dom->refs == 0) { - virDomainObjUnlock(dom); - virDomainObjFree(dom); - return 0; - } - return dom->refs; + virObjectUnref(&dom->obj); }
static virDomainObjPtr virDomainObjNew(virCapsPtr caps) @@ -1010,6 +1007,11 @@ static virDomainObjPtr virDomainObjNew(virCapsPtr caps) return NULL; }
+ if (virObjectInit(&domain->obj, doDomainObjFree)) { + VIR_FREE(domain); + return NULL; + } + if (caps->privateDataAllocFunc && !(domain->privateData = (caps->privateDataAllocFunc)())) { virReportOOMError(); @@ -1027,9 +1029,7 @@ static virDomainObjPtr virDomainObjNew(virCapsPtr caps) return NULL; }
- virDomainObjLock(domain); domain->state = VIR_DOMAIN_SHUTOFF; - domain->refs = 1;
virDomainSnapshotObjListInit(&domain->snapshots);
@@ -1075,8 +1075,10 @@ virDomainObjPtr virDomainAssignDef(virCapsPtr caps, domain->def = def;
virUUIDFormat(def->uuid, uuidstr); + virDomainObjRef(domain); if (virHashAddEntry(doms->objs, uuidstr, domain) < 0) { - VIR_FREE(domain); + virDomainObjUnref(domain); + virDomainObjFree(domain); return NULL; }
Simiarly here, you're now requiring all callers to manually obtain a lock.
That is the desired result, lock right before do read/write and unlock it as soon as possible.
@@ -1239,39 +1243,57 @@ static virDomainPtr qemudDomainCreate(virConnectPtr conn, const char *xml,
qemuDriverLock(driver); if (!(def = virDomainDefParseString(driver->caps, xml, - VIR_DOMAIN_XML_INACTIVE))) + VIR_DOMAIN_XML_INACTIVE))) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (virSecurityManagerVerify(driver->securityManager, def) < 0) + if (virSecurityManagerVerify(driver->securityManager, def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (virDomainObjIsDuplicate(&driver->domains, def, 1) < 0) + if (virDomainObjIsDuplicate(&driver->domains, def, 1) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (qemudCanonicalizeMachine(driver, def) < 0) + if (qemudCanonicalizeMachine(driver, def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
- if (qemuDomainAssignPCIAddresses(def) < 0) + if (qemuDomainAssignPCIAddresses(def) < 0) { + qemuDriverUnlock(driver); goto cleanup; + }
if (!(vm = virDomainAssignDef(driver->caps, &driver->domains, - def, false))) + def, false))) { + qemuDriverUnlock(driver); goto cleanup; + } + + qemuDriverUnlock(driver);
driver is now unlocked....
def = NULL;
- if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) { + virDomainObjUnref(vm);
...but this method *requires* driver to be locked.
goto cleanup; /* XXXX free the 'vm' we created ? */ + }
if (qemuProcessStart(conn, driver, vm, NULL, (flags & VIR_DOMAIN_START_PAUSED) != 0, -1, NULL, VIR_VM_OP_CREATE) < 0) { qemuAuditDomainStart(vm, "booted", false);
...and this method writes to 'driver', so it is now unsafe.
- if (qemuDomainObjEndJob(vm) > 0) - virDomainRemoveInactive(&driver->domains, - vm); - vm = NULL; + qemuDomainObjEndJob(vm); + qemuDriverLock(driver); + virDomainRemoveInactive(&driver->domains, + vm); + qemuDriverUnlock(driver); + virDomainObjUnref(vm); goto cleanup; }
@@ -1283,17 +1305,13 @@ static virDomainPtr qemudDomainCreate(virConnectPtr conn, const char *xml, dom = virGetDomain(conn, vm->def->name, vm->def->uuid); if (dom) dom->id = vm->def->id;
- if (vm && - qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm); + virDomainObjUnref(vm);
cleanup: virDomainDefFree(def); - if (vm) - virDomainObjUnlock(vm); if (event) qemuDomainEventQueue(driver, event); - qemuDriverUnlock(driver); return dom; }
And all the usage of 'vm' in this method is now completely unlocked and unsafe.
@@ -1307,13 +1325,14 @@ static int qemudDomainSuspend(virDomainPtr dom) {
qemuDriverLock(driver); vm = virDomainFindByUUID(&driver->domains, dom->uuid); + qemuDriverUnlock(driver);
if (!vm) { char uuidstr[VIR_UUID_STRING_BUFLEN]; virUUIDFormat(dom->uuid, uuidstr); qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); - goto cleanup; + return -1; } if (!virDomainObjIsActive(vm)) { qemuReportError(VIR_ERR_OPERATION_INVALID, @@ -1354,16 +1373,12 @@ static int qemudDomainSuspend(virDomainPtr dom) { }
endjob: - if (qemuDomainObjEndJob(vm) == 0) - vm = NULL; + qemuDomainObjEndJob(vm);
cleanup: - if (vm) - virDomainObjUnlock(vm); - + virDomainObjUnref(vm); if (event) qemuDomainEventQueue(driver, event); - qemuDriverUnlock(driver); return ret; }
Also now completely unsafe since 'vm' is never locked while making changes to it.
Yes. Will improve qemudDomainSuspend.
[cut rest of patch]
I don't see how any of this patch is threadsafe now that virtually no methods are acquiring the 'vm' lock.
qemuDomainObjBeginJob does acquire vm lock.
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

--- src/qemu/qemu_domain.c | 30 ++----------- src/qemu/qemu_migration.c | 2 - src/qemu/qemu_monitor.c | 109 ++++++++++++++++++++++++-------------------- src/qemu/qemu_monitor.h | 4 +- src/qemu/qemu_process.c | 32 +++++++------- 5 files changed, 81 insertions(+), 96 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 3a3c953..d11dc5f 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -560,9 +560,8 @@ void qemuDomainObjEnterMonitor(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - qemuMonitorLock(priv->mon); - qemuMonitorRef(priv->mon); virDomainObjUnlock(obj); + qemuMonitorLock(priv->mon); } @@ -573,18 +572,9 @@ void qemuDomainObjEnterMonitor(virDomainObjPtr obj) void qemuDomainObjExitMonitor(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - int refs; - - refs = qemuMonitorUnref(priv->mon); - - if (refs > 0) - qemuMonitorUnlock(priv->mon); + qemuMonitorUnlock(priv->mon); virDomainObjLock(obj); - - if (refs == 0) { - priv->mon = NULL; - } } @@ -601,10 +591,8 @@ void qemuDomainObjEnterMonitorWithDriver(struct qemud_driver *driver, { qemuDomainObjPrivatePtr priv = obj->privateData; - qemuMonitorLock(priv->mon); - qemuMonitorRef(priv->mon); virDomainObjUnlock(obj); - qemuDriverUnlock(driver); + qemuMonitorLock(priv->mon); } @@ -617,19 +605,9 @@ void qemuDomainObjExitMonitorWithDriver(struct qemud_driver *driver, virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - int refs; - - refs = qemuMonitorUnref(priv->mon); - - if (refs > 0) - qemuMonitorUnlock(priv->mon); - qemuDriverLock(driver); + qemuMonitorUnlock(priv->mon); virDomainObjLock(obj); - - if (refs == 0) { - priv->mon = NULL; - } } void qemuDomainObjEnterRemoteWithDriver(struct qemud_driver *driver, diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 462e6be..6af2e24 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -224,11 +224,9 @@ qemuMigrationWaitForCompletion(struct qemud_driver *driver, virDomainObjPtr vm) } virDomainObjUnlock(vm); - qemuDriverUnlock(driver); nanosleep(&ts, NULL); - qemuDriverLock(driver); virDomainObjLock(vm); } diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index 2d28f8d..98c55e7 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -37,6 +37,7 @@ #include "memory.h" #include "logging.h" #include "files.h" +#include "virobject.h" #define VIR_FROM_THIS VIR_FROM_QEMU @@ -44,11 +45,10 @@ #define DEBUG_RAW_IO 0 struct _qemuMonitor { + virObject obj; virMutex lock; /* also used to protect fd */ virCond notify; - int refs; - int fd; int watch; int hasSendFD; @@ -203,35 +203,61 @@ void qemuMonitorUnlock(qemuMonitorPtr mon) } -static void qemuMonitorFree(qemuMonitorPtr mon) +static void doMonitorFree(virObjectPtr obj) { + qemuMonitorPtr mon = (qemuMonitorPtr)obj; + VIR_DEBUG("mon=%p", mon); if (mon->cb && mon->cb->destroy) (mon->cb->destroy)(mon, mon->vm); - if (virCondDestroy(&mon->notify) < 0) - {} + ignore_value(virCondDestroy(&mon->notify)); virMutexDestroy(&mon->lock); VIR_FREE(mon->buffer); VIR_FREE(mon); } -int qemuMonitorRef(qemuMonitorPtr mon) +static qemuMonitorPtr qemuMonitorAlloc(void) { - mon->refs++; - return mon->refs; -} + qemuMonitorPtr mon; -int qemuMonitorUnref(qemuMonitorPtr mon) -{ - mon->refs--; + if (VIR_ALLOC(mon) < 0) { + virReportOOMError(); + return NULL; + } - if (mon->refs == 0) { - qemuMonitorUnlock(mon); - qemuMonitorFree(mon); - return 0; + if (virObjectInit(&mon->obj, doMonitorFree)) { + VIR_FREE(mon); + return NULL; + } + + if (virMutexInit(&mon->lock) < 0) { + qemuReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("cannot initialize monitor mutex")); + VIR_FREE(mon); + return NULL; } - return mon->refs; + if (virCondInit(&mon->notify) < 0) { + qemuReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("cannot initialize monitor condition")); + virMutexDestroy(&mon->lock); + VIR_FREE(mon); + return NULL; + } + + mon->fd = -1; + + return mon; +} + +void qemuMonitorRef(qemuMonitorPtr mon) +{ + virObjectRef(&mon->obj); +} + +void qemuMonitorUnref(qemuMonitorPtr mon) +{ + virObjectUnref(&mon->obj); } static void @@ -239,9 +265,7 @@ qemuMonitorUnwatch(void *monitor) { qemuMonitorPtr mon = monitor; - qemuMonitorLock(mon); - if (qemuMonitorUnref(mon) > 0) - qemuMonitorUnlock(mon); + qemuMonitorUnref(mon); } static int @@ -521,7 +545,6 @@ qemuMonitorIO(int watch, int fd, int events, void *opaque) { /* lock access to the monitor and protect fd */ qemuMonitorLock(mon); - qemuMonitorRef(mon); #if DEBUG_IO VIR_DEBUG("Monitor %p I/O on watch %d fd %d events %d", mon, watch, fd, events); #endif @@ -598,13 +621,11 @@ qemuMonitorIO(int watch, int fd, int events, void *opaque) { /* Make sure anyone waiting wakes up now */ virCondSignal(&mon->notify); - if (qemuMonitorUnref(mon) > 0) - qemuMonitorUnlock(mon); + qemuMonitorUnlock(mon); VIR_DEBUG("Triggering EOF callback error? %d", failed); (eofNotify)(mon, vm, failed); } else { - if (qemuMonitorUnref(mon) > 0) - qemuMonitorUnlock(mon); + qemuMonitorUnlock(mon); } } @@ -623,30 +644,13 @@ qemuMonitorOpen(virDomainObjPtr vm, return NULL; } - if (VIR_ALLOC(mon) < 0) { - virReportOOMError(); + mon = qemuMonitorAlloc(); + if (!mon) return NULL; - } - if (virMutexInit(&mon->lock) < 0) { - qemuReportError(VIR_ERR_INTERNAL_ERROR, "%s", - _("cannot initialize monitor mutex")); - VIR_FREE(mon); - return NULL; - } - if (virCondInit(&mon->notify) < 0) { - qemuReportError(VIR_ERR_INTERNAL_ERROR, "%s", - _("cannot initialize monitor condition")); - virMutexDestroy(&mon->lock); - VIR_FREE(mon); - return NULL; - } - mon->fd = -1; - mon->refs = 1; mon->vm = vm; mon->json = json; mon->cb = cb; - qemuMonitorLock(mon); switch (config->type) { case VIR_DOMAIN_CHR_TYPE_UNIX: @@ -679,20 +683,25 @@ qemuMonitorOpen(virDomainObjPtr vm, } + /* mon will be accessed by qemuMonitorIO which is called in + * event thread, so ref it before passing it to the thread. + * + * Note: mon is unrefed in qemuMonitorUnwatch + */ + qemuMonitorRef(mon); if ((mon->watch = virEventAddHandle(mon->fd, VIR_EVENT_HANDLE_HANGUP | VIR_EVENT_HANDLE_ERROR | VIR_EVENT_HANDLE_READABLE, qemuMonitorIO, mon, qemuMonitorUnwatch)) < 0) { + qemuMonitorUnref(mon); qemuReportError(VIR_ERR_INTERNAL_ERROR, "%s", _("unable to register monitor events")); goto cleanup; } - qemuMonitorRef(mon); VIR_DEBUG("New mon %p fd =%d watch=%d", mon, mon->fd, mon->watch); - qemuMonitorUnlock(mon); return mon; @@ -703,8 +712,8 @@ cleanup: * so kill the callbacks now. */ mon->cb = NULL; - qemuMonitorUnlock(mon); qemuMonitorClose(mon); + qemuMonitorUnref(mon); return NULL; } @@ -724,8 +733,8 @@ void qemuMonitorClose(qemuMonitorPtr mon) VIR_FORCE_CLOSE(mon->fd); } - if (qemuMonitorUnref(mon) > 0) - qemuMonitorUnlock(mon); + qemuMonitorUnlock(mon); + qemuMonitorUnref(mon); } @@ -778,7 +787,7 @@ int qemuMonitorHMPCommandWithFd(qemuMonitorPtr mon, if ((mon)->cb && (mon)->cb->callback) \ (ret) = ((mon)->cb->callback)(mon, __VA_ARGS__); \ qemuMonitorLock(mon); \ - ignore_value(qemuMonitorUnref(mon)); \ + qemuMonitorUnref(mon); \ } while (0) int qemuMonitorGetDiskSecret(qemuMonitorPtr mon, diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index c90219b..4498c49 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -132,8 +132,8 @@ int qemuMonitorCheckHMP(qemuMonitorPtr mon, const char *cmd); void qemuMonitorLock(qemuMonitorPtr mon); void qemuMonitorUnlock(qemuMonitorPtr mon); -int qemuMonitorRef(qemuMonitorPtr mon); -int qemuMonitorUnref(qemuMonitorPtr mon) ATTRIBUTE_RETURN_CHECK; +void qemuMonitorRef(qemuMonitorPtr mon); +void qemuMonitorUnref(qemuMonitorPtr mon); /* These APIs are for use by the internal Text/JSON monitor impl code only */ int qemuMonitorSend(qemuMonitorPtr mon, diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 244b22a..4b9087f 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -107,7 +107,6 @@ qemuProcessHandleMonitorEOF(qemuMonitorPtr mon ATTRIBUTE_UNUSED, VIR_DEBUG("Received EOF on %p '%s'", vm, vm->def->name); - qemuDriverLock(driver); virDomainObjLock(vm); if (!virDomainObjIsActive(vm)) { @@ -133,15 +132,17 @@ qemuProcessHandleMonitorEOF(qemuMonitorPtr mon ATTRIBUTE_UNUSED, qemuProcessStop(driver, vm, 0); qemuAuditDomainStop(vm, hasError ? "failed" : "shutdown"); - if (!vm->persistent) + if (!vm->persistent) { + qemuDriverLock(driver); virDomainRemoveInactive(&driver->domains, vm); - else - virDomainObjUnlock(vm); + qemuDriverUnlock(driver); + } + + virDomainObjUnlock(vm); if (event) { qemuDomainEventQueue(driver, event); } - qemuDriverUnlock(driver); } @@ -602,8 +603,10 @@ static void qemuProcessHandleMonitorDestroy(qemuMonitorPtr mon, virDomainObjLock(vm); priv = vm->privateData; - if (priv->mon == mon) + if (priv->mon == mon) { priv->mon = NULL; + qemuMonitorUnref(mon); + } virDomainObjUnlock(vm); virDomainObjUnref(vm); } @@ -633,19 +636,11 @@ qemuConnectMonitor(struct qemud_driver *driver, virDomainObjPtr vm) goto error; } - /* Hold an extra reference because we can't allow 'vm' to be - * deleted while the monitor is active */ - virDomainObjRef(vm); - priv->mon = qemuMonitorOpen(vm, priv->monConfig, priv->monJSON, &monitorCallbacks); - /* Safe to ignore value since ref count was incremented above */ - if (priv->mon == NULL) - virDomainObjUnref(vm); - if (virSecurityManagerClearSocketLabel(driver->securityManager, vm) < 0) { VIR_ERROR(_("Failed to clear security context for monitor for %s"), vm->def->name); @@ -657,6 +652,7 @@ qemuConnectMonitor(struct qemud_driver *driver, virDomainObjPtr vm) goto error; } + qemuMonitorRef(priv->mon); qemuDomainObjEnterMonitorWithDriver(driver, vm); ret = qemuMonitorSetCapabilities(priv->mon); @@ -2471,8 +2467,12 @@ void qemuProcessStop(struct qemud_driver *driver, _("Failed to send SIGTERM to %s (%d)"), vm->def->name, vm->pid); - if (priv->mon) - qemuMonitorClose(priv->mon); + if (priv->mon) { + qemuMonitorPtr mon = priv->mon; + priv->mon = NULL; + qemuMonitorClose(mon); + qemuMonitorUnref(mon); + } if (priv->monConfig) { if (priv->monConfig->type == VIR_DOMAIN_CHR_TYPE_UNIX) -- 1.7.3.1 -- Thanks, Hu Tao

On Wed, Apr 06, 2011 at 03:19:55PM +0800, Hu Tao wrote:
--- src/qemu/qemu_domain.c | 30 ++----------- src/qemu/qemu_migration.c | 2 - src/qemu/qemu_monitor.c | 109 ++++++++++++++++++++++++-------------------- src/qemu/qemu_monitor.h | 4 +- src/qemu/qemu_process.c | 32 +++++++------- 5 files changed, 81 insertions(+), 96 deletions(-)
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 3a3c953..d11dc5f 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -560,9 +560,8 @@ void qemuDomainObjEnterMonitor(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData;
- qemuMonitorLock(priv->mon); - qemuMonitorRef(priv->mon); virDomainObjUnlock(obj); + qemuMonitorLock(priv->mon); }
@@ -573,18 +572,9 @@ void qemuDomainObjEnterMonitor(virDomainObjPtr obj) void qemuDomainObjExitMonitor(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - int refs; - - refs = qemuMonitorUnref(priv->mon); - - if (refs > 0) - qemuMonitorUnlock(priv->mon);
+ qemuMonitorUnlock(priv->mon); virDomainObjLock(obj); - - if (refs == 0) { - priv->mon = NULL; - } }
@@ -601,10 +591,8 @@ void qemuDomainObjEnterMonitorWithDriver(struct qemud_driver *driver, { qemuDomainObjPrivatePtr priv = obj->privateData;
- qemuMonitorLock(priv->mon); - qemuMonitorRef(priv->mon); virDomainObjUnlock(obj); - qemuDriverUnlock(driver); + qemuMonitorLock(priv->mon); }
@@ -617,19 +605,9 @@ void qemuDomainObjExitMonitorWithDriver(struct qemud_driver *driver, virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - int refs; - - refs = qemuMonitorUnref(priv->mon); - - if (refs > 0) - qemuMonitorUnlock(priv->mon);
- qemuDriverLock(driver); + qemuMonitorUnlock(priv->mon); virDomainObjLock(obj); - - if (refs == 0) { - priv->mon = NULL; - } }
This means that the 'driver' lock is now whenever any QEMU monitor command is runing, which blocks the entire driver.
void qemuDomainObjEnterRemoteWithDriver(struct qemud_driver *driver, diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 462e6be..6af2e24 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -224,11 +224,9 @@ qemuMigrationWaitForCompletion(struct qemud_driver *driver, virDomainObjPtr vm) }
virDomainObjUnlock(vm); - qemuDriverUnlock(driver);
nanosleep(&ts, NULL);
- qemuDriverLock(driver); virDomainObjLock(vm); }
Holding the 'driver' lock while sleeping blocks the entire QEMU driver.
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 244b22a..4b9087f 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -107,7 +107,6 @@ qemuProcessHandleMonitorEOF(qemuMonitorPtr mon ATTRIBUTE_UNUSED,
VIR_DEBUG("Received EOF on %p '%s'", vm, vm->def->name);
- qemuDriverLock(driver); virDomainObjLock(vm);
if (!virDomainObjIsActive(vm)) { @@ -133,15 +132,17 @@ qemuProcessHandleMonitorEOF(qemuMonitorPtr mon ATTRIBUTE_UNUSED, qemuProcessStop(driver, vm, 0); qemuAuditDomainStop(vm, hasError ? "failed" : "shutdown");
- if (!vm->persistent) + if (!vm->persistent) { + qemuDriverLock(driver); virDomainRemoveInactive(&driver->domains, vm); - else - virDomainObjUnlock(vm); + qemuDriverUnlock(driver); + } + + virDomainObjUnlock(vm);
if (event) { qemuDomainEventQueue(driver, event); } - qemuDriverUnlock(driver); }
This violates the lock ordering rules. The 'driver' lock *must* be obtained *before* any 'vm' lock is held. Now we have some places in the code which do lock(vm) lock(driver) and other places which do lock(driver) lock(vm) so 2 threads can trivially deadlock waiting for each other Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Thu, Apr 07, 2011 at 10:38:34AM +0100, Daniel P. Berrange wrote:
On Wed, Apr 06, 2011 at 03:19:55PM +0800, Hu Tao wrote:
--- src/qemu/qemu_domain.c | 30 ++----------- src/qemu/qemu_migration.c | 2 - src/qemu/qemu_monitor.c | 109 ++++++++++++++++++++++++-------------------- src/qemu/qemu_monitor.h | 4 +- src/qemu/qemu_process.c | 32 +++++++------- 5 files changed, 81 insertions(+), 96 deletions(-)
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 3a3c953..d11dc5f 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -560,9 +560,8 @@ void qemuDomainObjEnterMonitor(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData;
- qemuMonitorLock(priv->mon); - qemuMonitorRef(priv->mon); virDomainObjUnlock(obj); + qemuMonitorLock(priv->mon); }
@@ -573,18 +572,9 @@ void qemuDomainObjEnterMonitor(virDomainObjPtr obj) void qemuDomainObjExitMonitor(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - int refs; - - refs = qemuMonitorUnref(priv->mon); - - if (refs > 0) - qemuMonitorUnlock(priv->mon);
+ qemuMonitorUnlock(priv->mon); virDomainObjLock(obj); - - if (refs == 0) { - priv->mon = NULL; - } }
@@ -601,10 +591,8 @@ void qemuDomainObjEnterMonitorWithDriver(struct qemud_driver *driver, { qemuDomainObjPrivatePtr priv = obj->privateData;
- qemuMonitorLock(priv->mon); - qemuMonitorRef(priv->mon); virDomainObjUnlock(obj); - qemuDriverUnlock(driver); + qemuMonitorLock(priv->mon); }
@@ -617,19 +605,9 @@ void qemuDomainObjExitMonitorWithDriver(struct qemud_driver *driver, virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - int refs; - - refs = qemuMonitorUnref(priv->mon); - - if (refs > 0) - qemuMonitorUnlock(priv->mon);
- qemuDriverLock(driver); + qemuMonitorUnlock(priv->mon); virDomainObjLock(obj); - - if (refs == 0) { - priv->mon = NULL; - } }
This means that the 'driver' lock is now whenever any QEMU monitor command is runing, which blocks the entire driver.
qemuDomainObjEnterMonitorWithDriver is now called without holding qemu driver lock.
void qemuDomainObjEnterRemoteWithDriver(struct qemud_driver *driver, diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 462e6be..6af2e24 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -224,11 +224,9 @@ qemuMigrationWaitForCompletion(struct qemud_driver *driver, virDomainObjPtr vm) }
virDomainObjUnlock(vm); - qemuDriverUnlock(driver);
nanosleep(&ts, NULL);
- qemuDriverLock(driver); virDomainObjLock(vm); }
Holding the 'driver' lock while sleeping blocks the entire QEMU driver.
Now qemuMigrationWaitForCompletion should be called without holding qemu driver lock.
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 244b22a..4b9087f 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -107,7 +107,6 @@ qemuProcessHandleMonitorEOF(qemuMonitorPtr mon ATTRIBUTE_UNUSED,
VIR_DEBUG("Received EOF on %p '%s'", vm, vm->def->name);
- qemuDriverLock(driver); virDomainObjLock(vm);
if (!virDomainObjIsActive(vm)) { @@ -133,15 +132,17 @@ qemuProcessHandleMonitorEOF(qemuMonitorPtr mon ATTRIBUTE_UNUSED, qemuProcessStop(driver, vm, 0); qemuAuditDomainStop(vm, hasError ? "failed" : "shutdown");
- if (!vm->persistent) + if (!vm->persistent) { + qemuDriverLock(driver); virDomainRemoveInactive(&driver->domains, vm); - else - virDomainObjUnlock(vm); + qemuDriverUnlock(driver); + } + + virDomainObjUnlock(vm);
if (event) { qemuDomainEventQueue(driver, event); } - qemuDriverUnlock(driver); }
This violates the lock ordering rules. The 'driver' lock *must* be obtained *before* any 'vm' lock is held.
Excepting for introducing virObject for reference-counting, this series also simplifies the usage of lock: if you want to read/write qemu driver data, it is enough to first acquire qemu driver lock only; if you want to read/write virDomainObj data, it is enough to first acquire virDomainObj lock only; same for others. And we'd better to avoid acquiring two locks at the same time. So yes, the code here is problematic, it should be ideally like this: virDomainObjLock(vm); if (!vm->persistent) { lock_hashtable(doms); /* hashtable's own lock to protect itself */ virDomainRemoveInactive(doms, vm); unlock_hashtable(doms); } virDomainObjUnlock(vm); But it lacks hashtable lock, how about change the code like this: virDomainObjLock(vm); persistent = vm->persistent; virDomainObjUnlock(vm); /* chances that others change vm->persistent and we remove vm mistakenly :( */ if (!persistent) { qemuDriverLock(driver); virDomainRemoveInactive(doms, vm); qemuDriverUnlock(driver); } Or is there a better way?
Now we have some places in the code which do
lock(vm) lock(driver)
and other places which do
lock(driver) lock(vm)
so 2 threads can trivially deadlock waiting for each other
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Fri, Apr 08, 2011 at 11:07:44AM +0800, Hu Tao wrote:
On Thu, Apr 07, 2011 at 10:38:34AM +0100, Daniel P. Berrange wrote:
On Wed, Apr 06, 2011 at 03:19:55PM +0800, Hu Tao wrote:
--- src/qemu/qemu_domain.c | 30 ++----------- src/qemu/qemu_migration.c | 2 - src/qemu/qemu_monitor.c | 109 ++++++++++++++++++++++++-------------------- src/qemu/qemu_monitor.h | 4 +- src/qemu/qemu_process.c | 32 +++++++------- 5 files changed, 81 insertions(+), 96 deletions(-)
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 3a3c953..d11dc5f 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -560,9 +560,8 @@ void qemuDomainObjEnterMonitor(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData;
- qemuMonitorLock(priv->mon); - qemuMonitorRef(priv->mon); virDomainObjUnlock(obj); + qemuMonitorLock(priv->mon); }
@@ -573,18 +572,9 @@ void qemuDomainObjEnterMonitor(virDomainObjPtr obj) void qemuDomainObjExitMonitor(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - int refs; - - refs = qemuMonitorUnref(priv->mon); - - if (refs > 0) - qemuMonitorUnlock(priv->mon);
+ qemuMonitorUnlock(priv->mon); virDomainObjLock(obj); - - if (refs == 0) { - priv->mon = NULL; - } }
@@ -601,10 +591,8 @@ void qemuDomainObjEnterMonitorWithDriver(struct qemud_driver *driver, { qemuDomainObjPrivatePtr priv = obj->privateData;
- qemuMonitorLock(priv->mon); - qemuMonitorRef(priv->mon); virDomainObjUnlock(obj); - qemuDriverUnlock(driver); + qemuMonitorLock(priv->mon); }
@@ -617,19 +605,9 @@ void qemuDomainObjExitMonitorWithDriver(struct qemud_driver *driver, virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - int refs; - - refs = qemuMonitorUnref(priv->mon); - - if (refs > 0) - qemuMonitorUnlock(priv->mon);
- qemuDriverLock(driver); + qemuMonitorUnlock(priv->mon); virDomainObjLock(obj); - - if (refs == 0) { - priv->mon = NULL; - } }
This means that the 'driver' lock is now whenever any QEMU monitor command is runing, which blocks the entire driver.
qemuDomainObjEnterMonitorWithDriver is now called without holding qemu driver lock.
Well, this and the other changes in this series are completely altering all the locking rules used throughout the QEMU driver, with no clear explanation of what you are actually doing. Please read src/qemu/THREADS.txt and then provide an equivalent document explaining what you think the new rules should be, otherwise it is impossible to tell if these patches are at all threadsafe. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Fri, Apr 08, 2011 at 09:57:16AM +0100, Daniel P. Berrange wrote:
On Fri, Apr 08, 2011 at 11:07:44AM +0800, Hu Tao wrote:
On Thu, Apr 07, 2011 at 10:38:34AM +0100, Daniel P. Berrange wrote:
On Wed, Apr 06, 2011 at 03:19:55PM +0800, Hu Tao wrote:
--- src/qemu/qemu_domain.c | 30 ++----------- src/qemu/qemu_migration.c | 2 - src/qemu/qemu_monitor.c | 109 ++++++++++++++++++++++++-------------------- src/qemu/qemu_monitor.h | 4 +- src/qemu/qemu_process.c | 32 +++++++------- 5 files changed, 81 insertions(+), 96 deletions(-)
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 3a3c953..d11dc5f 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -560,9 +560,8 @@ void qemuDomainObjEnterMonitor(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData;
- qemuMonitorLock(priv->mon); - qemuMonitorRef(priv->mon); virDomainObjUnlock(obj); + qemuMonitorLock(priv->mon); }
@@ -573,18 +572,9 @@ void qemuDomainObjEnterMonitor(virDomainObjPtr obj) void qemuDomainObjExitMonitor(virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - int refs; - - refs = qemuMonitorUnref(priv->mon); - - if (refs > 0) - qemuMonitorUnlock(priv->mon);
+ qemuMonitorUnlock(priv->mon); virDomainObjLock(obj); - - if (refs == 0) { - priv->mon = NULL; - } }
@@ -601,10 +591,8 @@ void qemuDomainObjEnterMonitorWithDriver(struct qemud_driver *driver, { qemuDomainObjPrivatePtr priv = obj->privateData;
- qemuMonitorLock(priv->mon); - qemuMonitorRef(priv->mon); virDomainObjUnlock(obj); - qemuDriverUnlock(driver); + qemuMonitorLock(priv->mon); }
@@ -617,19 +605,9 @@ void qemuDomainObjExitMonitorWithDriver(struct qemud_driver *driver, virDomainObjPtr obj) { qemuDomainObjPrivatePtr priv = obj->privateData; - int refs; - - refs = qemuMonitorUnref(priv->mon); - - if (refs > 0) - qemuMonitorUnlock(priv->mon);
- qemuDriverLock(driver); + qemuMonitorUnlock(priv->mon); virDomainObjLock(obj); - - if (refs == 0) { - priv->mon = NULL; - } }
This means that the 'driver' lock is now whenever any QEMU monitor command is runing, which blocks the entire driver.
qemuDomainObjEnterMonitorWithDriver is now called without holding qemu driver lock.
Well, this and the other changes in this series are completely altering all the locking rules used throughout the QEMU driver, with no clear explanation of what you are actually doing. Please read src/qemu/THREADS.txt and then provide an equivalent document explaining what you think the new rules should be, otherwise it is impossible to tell if these patches are at all threadsafe.
Sorry I didn't make this clear, will do in next version.

Bodies of qemuDomainObjEnterMonitorWithDriver/qemuDomainObjExitMonitorWithDriver are the same as qemuDomainObjEnterMonitor/qemuDomainObjExitMonitor, so remove them. --- src/qemu/qemu_domain.c | 32 ---------------- src/qemu/qemu_domain.h | 4 -- src/qemu/qemu_driver.c | 20 +++++----- src/qemu/qemu_hotplug.c | 90 ++++++++++++++++++++++---------------------- src/qemu/qemu_migration.c | 46 +++++++++++----------- src/qemu/qemu_process.c | 36 +++++++++--------- 6 files changed, 96 insertions(+), 132 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index d11dc5f..b9bf8a4 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -578,38 +578,6 @@ void qemuDomainObjExitMonitor(virDomainObjPtr obj) } -/* - * obj must be locked before calling, qemud_driver must be locked - * - * To be called immediately before any QEMU monitor API call - * Must have already called qemuDomainObjBeginJob(). - * - * To be followed with qemuDomainObjExitMonitorWithDriver() once complete - */ -void qemuDomainObjEnterMonitorWithDriver(struct qemud_driver *driver, - virDomainObjPtr obj) -{ - qemuDomainObjPrivatePtr priv = obj->privateData; - - virDomainObjUnlock(obj); - qemuMonitorLock(priv->mon); -} - - -/* obj must NOT be locked before calling, qemud_driver must be unlocked, - * and will be locked after returning - * - * Should be paired with an earlier qemuDomainObjEnterMonitorWithDriver() call - */ -void qemuDomainObjExitMonitorWithDriver(struct qemud_driver *driver, - virDomainObjPtr obj) -{ - qemuDomainObjPrivatePtr priv = obj->privateData; - - qemuMonitorUnlock(priv->mon); - virDomainObjLock(obj); -} - void qemuDomainObjEnterRemoteWithDriver(struct qemud_driver *driver, virDomainObjPtr obj) { diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index 12fd21d..75ad5d9 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -100,10 +100,6 @@ int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, void qemuDomainObjEndJob(virDomainObjPtr obj); void qemuDomainObjEnterMonitor(virDomainObjPtr obj); void qemuDomainObjExitMonitor(virDomainObjPtr obj); -void qemuDomainObjEnterMonitorWithDriver(struct qemud_driver *driver, - virDomainObjPtr obj); -void qemuDomainObjExitMonitorWithDriver(struct qemud_driver *driver, - virDomainObjPtr obj); void qemuDomainObjEnterRemoteWithDriver(struct qemud_driver *driver, virDomainObjPtr obj); void qemuDomainObjExitRemoteWithDriver(struct qemud_driver *driver, diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index db89402..887a865 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -3251,9 +3251,9 @@ static char *qemudDomainDumpXML(virDomainPtr dom, if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) goto cleanup; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); err = qemuMonitorGetBalloonInfo(priv->mon, &balloon); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuDomainObjEndJob(vm); if (err < 0) goto cleanup; @@ -6026,9 +6026,9 @@ qemuDomainSnapshotCreateActive(virConnectPtr conn, } } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorCreateSnapshot(priv->mon, snap->def->name); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); cleanup: if (resume && virDomainObjIsActive(vm) && @@ -6359,9 +6359,9 @@ static int qemuDomainRevertToSnapshot(virDomainSnapshotPtr snapshot, if (virDomainObjIsActive(vm)) { priv = vm->privateData; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); rc = qemuMonitorLoadSnapshot(priv->mon, snap->def->name); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (rc < 0) goto endjob; } @@ -6478,10 +6478,10 @@ static int qemuDomainSnapshotDiscard(struct qemud_driver *driver, } else { priv = vm->privateData; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); /* we continue on even in the face of error */ qemuMonitorDeleteSnapshot(priv->mon, snap->def->name); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); } if (snap == vm->current_snapshot) { @@ -6689,9 +6689,9 @@ static int qemuDomainMonitorCommand(virDomainPtr domain, const char *cmd, if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) goto cleanup; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorArbitraryCommand(priv->mon, cmd, result, hmp); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuDomainObjEndJob(vm); cleanup: diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c index b03f774..c1e893b 100644 --- a/src/qemu/qemu_hotplug.c +++ b/src/qemu/qemu_hotplug.c @@ -91,7 +91,7 @@ int qemuDomainChangeEjectableMedia(struct qemud_driver *driver, goto error; priv = vm->privateData; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (disk->src) { const char *format = NULL; if (disk->type != VIR_DOMAIN_DISK_TYPE_DIR) { @@ -106,7 +106,7 @@ int qemuDomainChangeEjectableMedia(struct qemud_driver *driver, } else { ret = qemuMonitorEjectMedia(priv->mon, driveAlias, force); } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditDisk(vm, origdisk, disk, "update", ret >= 0); @@ -178,7 +178,7 @@ int qemuDomainAttachPciDiskDevice(struct qemud_driver *driver, goto error; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { ret = qemuMonitorAddDrive(priv->mon, drivestr); if (ret == 0) { @@ -201,7 +201,7 @@ int qemuDomainAttachPciDiskDevice(struct qemud_driver *driver, memcpy(&disk->info.addr.pci, &guestAddr, sizeof(guestAddr)); } } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditDisk(vm, NULL, disk, "attach", ret >= 0); @@ -269,7 +269,7 @@ int qemuDomainAttachPciControllerDevice(struct qemud_driver *driver, goto cleanup; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { ret = qemuMonitorAddDevice(priv->mon, devstr); } else { @@ -277,7 +277,7 @@ int qemuDomainAttachPciControllerDevice(struct qemud_driver *driver, type, &controller->info.addr.pci); } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (ret == 0) { controller->info.type = VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI; @@ -408,7 +408,7 @@ int qemuDomainAttachSCSIDisk(struct qemud_driver *driver, goto error; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { ret = qemuMonitorAddDrive(priv->mon, drivestr); if (ret == 0) { @@ -433,7 +433,7 @@ int qemuDomainAttachSCSIDisk(struct qemud_driver *driver, memcpy(&disk->info.addr.drive, &driveAddr, sizeof(driveAddr)); } } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditDisk(vm, NULL, disk, "attach", ret >= 0); @@ -501,7 +501,7 @@ int qemuDomainAttachUsbMassstorageDevice(struct qemud_driver *driver, goto error; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { ret = qemuMonitorAddDrive(priv->mon, drivestr); if (ret == 0) { @@ -516,7 +516,7 @@ int qemuDomainAttachUsbMassstorageDevice(struct qemud_driver *driver, } else { ret = qemuMonitorAddUSBDisk(priv->mon, disk->src); } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditDisk(vm, NULL, disk, "attach", ret >= 0); @@ -629,24 +629,24 @@ int qemuDomainAttachNetDevice(virConnectPtr conn, goto cleanup; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_NETDEV) && qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { if (qemuMonitorAddNetdev(priv->mon, netstr, tapfd, tapfd_name, vhostfd, vhostfd_name) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditNet(vm, NULL, net, "attach", false); goto cleanup; } } else { if (qemuMonitorAddHostNetwork(priv->mon, netstr, tapfd, tapfd_name, vhostfd, vhostfd_name) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditNet(vm, NULL, net, "attach", false); goto cleanup; } } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); VIR_FORCE_CLOSE(tapfd); VIR_FORCE_CLOSE(vhostfd); @@ -665,24 +665,24 @@ int qemuDomainAttachNetDevice(virConnectPtr conn, goto try_remove; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { if (qemuMonitorAddDevice(priv->mon, nicstr) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditNet(vm, NULL, net, "attach", false); goto try_remove; } } else { if (qemuMonitorAddPCINetwork(priv->mon, nicstr, &guestAddr) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditNet(vm, NULL, net, "attach", false); goto try_remove; } net->info.type = VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI; memcpy(&net->info.addr.pci, &guestAddr, sizeof(guestAddr)); } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditNet(vm, NULL, net, "attach", true); @@ -719,11 +719,11 @@ try_remove: char *netdev_name; if (virAsprintf(&netdev_name, "host%s", net->info.alias) < 0) goto no_memory; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuMonitorRemoveNetdev(priv->mon, netdev_name) < 0) VIR_WARN("Failed to remove network backend for netdev %s", netdev_name); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); VIR_FREE(netdev_name); } else { VIR_WARN0("Unable to remove network backend"); @@ -732,11 +732,11 @@ try_remove: char *hostnet_name; if (virAsprintf(&hostnet_name, "host%s", net->info.alias) < 0) goto no_memory; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuMonitorRemoveHostNetwork(priv->mon, vlan, hostnet_name) < 0) VIR_WARN("Failed to remove network backend for vlan %d, net %s", vlan, hostnet_name); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); VIR_FREE(hostnet_name); } goto cleanup; @@ -792,18 +792,18 @@ int qemuDomainAttachHostPciDevice(struct qemud_driver *driver, qemuCaps))) goto error; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorAddDeviceWithFd(priv->mon, devstr, configfd, configfd_name); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); } else { virDomainDevicePCIAddress guestAddr; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorAddPCIHostDevice(priv->mon, &hostdev->source.subsys.u.pci, &guestAddr); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); hostdev->info.type = VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI; memcpy(&hostdev->info.addr.pci, &guestAddr, sizeof(guestAddr)); @@ -879,14 +879,14 @@ int qemuDomainAttachHostUsbDevice(struct qemud_driver *driver, goto error; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) ret = qemuMonitorAddDevice(priv->mon, devstr); else ret = qemuMonitorAddUSBDeviceExact(priv->mon, hostdev->source.subsys.u.usb.bus, hostdev->source.subsys.u.usb.device); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditHostdev(vm, hostdev, "attach", ret == 0); if (ret < 0) goto error; @@ -1143,7 +1143,7 @@ int qemuDomainDetachPciDiskDevice(struct qemud_driver *driver, goto cleanup; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { if (qemuMonitorDelDevice(priv->mon, detach->info.alias) < 0) { qemuDomainObjExitMonitor(vm); @@ -1160,7 +1160,7 @@ int qemuDomainDetachPciDiskDevice(struct qemud_driver *driver, /* disconnect guest from host device */ qemuMonitorDriveDel(priv->mon, drivestr); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditDisk(vm, detach, NULL, "detach", ret >= 0); @@ -1234,7 +1234,7 @@ int qemuDomainDetachDiskDevice(struct qemud_driver *driver, goto cleanup; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuMonitorDelDevice(priv->mon, detach->info.alias) < 0) { qemuDomainObjExitMonitor(vm); goto cleanup; @@ -1243,7 +1243,7 @@ int qemuDomainDetachDiskDevice(struct qemud_driver *driver, /* disconnect guest from host device */ qemuMonitorDriveDel(priv->mon, drivestr); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditDisk(vm, detach, NULL, "detach", ret >= 0); @@ -1362,7 +1362,7 @@ int qemuDomainDetachPciControllerDevice(struct qemud_driver *driver, goto cleanup; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { if (qemuMonitorDelDevice(priv->mon, detach->info.alias)) { qemuDomainObjExitMonitor(vm); @@ -1375,7 +1375,7 @@ int qemuDomainDetachPciControllerDevice(struct qemud_driver *driver, goto cleanup; } } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (vm->def->ncontrollers > 1) { memmove(vm->def->controllers + i, @@ -1450,7 +1450,7 @@ int qemuDomainDetachNetDevice(struct qemud_driver *driver, goto cleanup; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { if (qemuMonitorDelDevice(priv->mon, detach->info.alias) < 0) { qemuDomainObjExitMonitor(vm); @@ -1460,7 +1460,7 @@ int qemuDomainDetachNetDevice(struct qemud_driver *driver, } else { if (qemuMonitorRemovePCIDevice(priv->mon, &detach->info.addr.pci) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditNet(vm, detach, NULL, "detach", false); goto cleanup; } @@ -1469,18 +1469,18 @@ int qemuDomainDetachNetDevice(struct qemud_driver *driver, if (qemuCapsGet(qemuCaps, QEMU_CAPS_NETDEV) && qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { if (qemuMonitorRemoveNetdev(priv->mon, hostnet_name) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditNet(vm, detach, NULL, "detach", false); goto cleanup; } } else { if (qemuMonitorRemoveHostNetwork(priv->mon, vlan, hostnet_name) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditNet(vm, detach, NULL, "detach", false); goto cleanup; } } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditNet(vm, detach, NULL, "detach", true); @@ -1576,13 +1576,13 @@ int qemuDomainDetachHostPciDevice(struct qemud_driver *driver, return -1; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuCapsGet(qemuCaps, QEMU_CAPS_DEVICE)) { ret = qemuMonitorDelDevice(priv->mon, detach->info.alias); } else { ret = qemuMonitorRemovePCIDevice(priv->mon, &detach->info.addr.pci); } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditHostdev(vm, detach, "detach", ret == 0); if (ret < 0) return -1; @@ -1679,9 +1679,9 @@ int qemuDomainDetachHostUsbDevice(struct qemud_driver *driver, return -1; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorDelDevice(priv->mon, detach->info.alias); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); qemuAuditHostdev(vm, detach, "detach", ret == 0); if (ret < 0) return -1; @@ -1755,7 +1755,7 @@ qemuDomainChangeGraphicsPasswords(struct qemud_driver *driver, if (!auth->passwd && !driver->vncPassword) return 0; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorSetPassword(priv->mon, type, auth->passwd ? auth->passwd : defaultPasswd, @@ -1798,7 +1798,7 @@ qemuDomainChangeGraphicsPasswords(struct qemud_driver *driver, } cleanup: - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); return ret; } diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 6af2e24..5076cd5 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -123,9 +123,9 @@ qemuMigrationWaitForCompletion(struct qemud_driver *driver, virDomainObjPtr vm) if (priv->jobSignals & QEMU_JOB_SIGNAL_CANCEL) { priv->jobSignals ^= QEMU_JOB_SIGNAL_CANCEL; VIR_DEBUG0("Cancelling job at client request"); - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); rc = qemuMonitorMigrateCancel(priv->mon); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (rc < 0) { VIR_WARN0("Unable to cancel job"); } @@ -140,9 +140,9 @@ qemuMigrationWaitForCompletion(struct qemud_driver *driver, virDomainObjPtr vm) priv->jobSignals ^= QEMU_JOB_SIGNAL_MIGRATE_DOWNTIME; priv->jobSignalsData.migrateDowntime = 0; VIR_DEBUG("Setting migration downtime to %llums", ms); - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); rc = qemuMonitorSetMigrationDowntime(priv->mon, ms); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (rc < 0) VIR_WARN0("Unable to set migration downtime"); } else if (priv->jobSignals & QEMU_JOB_SIGNAL_MIGRATE_SPEED) { @@ -151,9 +151,9 @@ qemuMigrationWaitForCompletion(struct qemud_driver *driver, virDomainObjPtr vm) priv->jobSignals ^= QEMU_JOB_SIGNAL_MIGRATE_SPEED; priv->jobSignalsData.migrateBandwidth = 0; VIR_DEBUG("Setting migration bandwidth to %luMbs", bandwidth); - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); rc = qemuMonitorSetMigrationSpeed(priv->mon, bandwidth); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (rc < 0) VIR_WARN0("Unable to set migration speed"); } @@ -167,13 +167,13 @@ qemuMigrationWaitForCompletion(struct qemud_driver *driver, virDomainObjPtr vm) goto cleanup; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); rc = qemuMonitorGetMigrationStatus(priv->mon, &status, &memProcessed, &memRemaining, &memTotal); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (rc < 0) { priv->jobInfo.type = VIR_DOMAIN_JOB_FAILED; @@ -602,10 +602,10 @@ static int doNativeMigrate(struct qemud_driver *driver, goto cleanup; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (resource > 0 && qemuMonitorSetMigrationSpeed(priv->mon, resource) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); goto cleanup; } @@ -617,10 +617,10 @@ static int doNativeMigrate(struct qemud_driver *driver, if (qemuMonitorMigrateToHost(priv->mon, background_flags, uribits->server, uribits->port) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); goto cleanup; } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (qemuMigrationWaitForCompletion(driver, vm) < 0) goto cleanup; @@ -804,7 +804,7 @@ static int doTunnelMigrate(struct qemud_driver *driver, } /* 3. start migration on source */ - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (flags & VIR_MIGRATE_NON_SHARED_DISK) background_flags |= QEMU_MONITOR_MIGRATE_NON_SHARED_DISK; if (flags & VIR_MIGRATE_NON_SHARED_INC) @@ -819,7 +819,7 @@ static int doTunnelMigrate(struct qemud_driver *driver, } else { internalret = -1; } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (internalret < 0) { qemuReportError(VIR_ERR_OPERATION_FAILED, "%s", _("tunnelled migration monitor command failed")); @@ -838,16 +838,16 @@ static int doTunnelMigrate(struct qemud_driver *driver, /* it is also possible that the migrate didn't fail initially, but * rather failed later on. Check the output of "info migrate" */ - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuMonitorGetMigrationStatus(priv->mon, &status, &transferred, &remaining, &total) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); goto cancel; } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (status == QEMU_MONITOR_MIGRATION_STATUS_ERROR) { qemuReportError(VIR_ERR_OPERATION_FAILED, @@ -868,9 +868,9 @@ static int doTunnelMigrate(struct qemud_driver *driver, cancel: if (retval != 0 && virDomainObjIsActive(vm)) { - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); qemuMonitorMigrateCancel(priv->mon); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); } finish: @@ -1337,7 +1337,7 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, restoreLabel = true; } - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (!compressor) { const char *args[] = { "cat", NULL }; @@ -1365,11 +1365,11 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, if (virSetCloseExec(pipeFD[1]) < 0) { virReportSystemError(errno, "%s", _("Unable to set cloexec flag")); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); goto cleanup; } if (virCommandRunAsync(cmd, NULL) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); goto cleanup; } rc = qemuMonitorMigrateToFd(priv->mon, @@ -1384,7 +1384,7 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, args, path, offset); } } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (rc < 0) goto cleanup; diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 4b9087f..a2bd94c 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -654,9 +654,9 @@ qemuConnectMonitor(struct qemud_driver *driver, virDomainObjPtr vm) qemuMonitorRef(priv->mon); - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorSetCapabilities(priv->mon); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); error: @@ -1062,9 +1062,9 @@ qemuProcessWaitForMonitor(struct qemud_driver* driver, goto cleanup; priv = vm->privateData; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorGetPtyPaths(priv->mon, paths); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); VIR_DEBUG("qemuMonitorGetPtyPaths returned %i", ret); if (ret == 0) @@ -1115,12 +1115,12 @@ qemuProcessDetectVcpuPIDs(struct qemud_driver *driver, /* What follows is now all KVM specific */ - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if ((ncpupids = qemuMonitorGetCPUInfo(priv->mon, &cpupids)) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); return -1; } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); /* Treat failure to get VCPU<->PID mapping as non-fatal */ if (ncpupids == 0) @@ -1313,10 +1313,10 @@ qemuProcessInitPasswords(virConnectPtr conn, goto cleanup; alias = vm->def->disks[i]->info.alias; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorSetDrivePassphrase(priv->mon, alias, secret); VIR_FREE(secret); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (ret < 0) goto cleanup; } @@ -1704,10 +1704,10 @@ qemuProcessInitPCIAddresses(struct qemud_driver *driver, int ret; qemuMonitorPCIAddress *addrs = NULL; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); naddrs = qemuMonitorGetAllPCIAddresses(priv->mon, &addrs); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); ret = qemuProcessDetectPCIAddresses(vm, addrs, naddrs); @@ -1860,9 +1860,9 @@ qemuProcessStartCPUs(struct qemud_driver *driver, virDomainObjPtr vm, int ret; qemuDomainObjPrivatePtr priv = vm->privateData; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorStartCPUs(priv->mon, conn); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (ret == 0) { vm->state = VIR_DOMAIN_RUNNING; } @@ -1878,9 +1878,9 @@ int qemuProcessStopCPUs(struct qemud_driver *driver, virDomainObjPtr vm) qemuDomainObjPrivatePtr priv = vm->privateData; vm->state = VIR_DOMAIN_PAUSED; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); ret = qemuMonitorStopCPUs(priv->mon); - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (ret < 0) { vm->state = oldState; } @@ -2351,12 +2351,12 @@ int qemuProcessStart(virConnectPtr conn, VIR_DEBUG0("Setting initial memory amount"); cur_balloon = vm->def->mem.cur_balloon; - qemuDomainObjEnterMonitorWithDriver(driver, vm); + qemuDomainObjEnterMonitor(vm); if (qemuMonitorSetBalloon(priv->mon, cur_balloon) < 0) { - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); goto cleanup; } - qemuDomainObjExitMonitorWithDriver(driver, vm); + qemuDomainObjExitMonitor(vm); if (!start_paused) { VIR_DEBUG0("Starting domain CPUs"); -- 1.7.3.1 -- Thanks, Hu Tao

On 04/06/2011 01:20 AM, Hu Tao wrote:
Bodies of qemuDomainObjEnterMonitorWithDriver/qemuDomainObjExitMonitorWithDriver are the same as qemuDomainObjEnterMonitor/qemuDomainObjExitMonitor, so remove them. --- src/qemu/qemu_domain.c | 32 ---------------- src/qemu/qemu_domain.h | 4 -- src/qemu/qemu_driver.c | 20 +++++----- src/qemu/qemu_hotplug.c | 90 ++++++++++++++++++++++---------------------- src/qemu/qemu_migration.c | 46 +++++++++++----------- src/qemu/qemu_process.c | 36 +++++++++--------- 6 files changed, 96 insertions(+), 132 deletions(-)
Aha - I should have previewed the patch series first, before raising the same point in my review of patch 3. I'm still not convinced that we can get away with this, if any of the callers had previously been calling with driver locked, because driver lock must be dropped before waiting for the condition variable. But if we can, then it's a nice concept. I've run out of review time for today, though, so I haven't reviewed this one closely (nor 4 or 6), and probably won't look at them until we resolve the bigger issues I raised earlier on patches 1 and 3 when you post a v2 of the series. Even documenting in the commit message of patch 3 that you are intentionally changing semantics of no longer calling with driver locked, but saving the cleanup for a later patch, would be helpful. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

The body of qemuDomainObjBeginJobWithDriver is the same as qemuDomainObjBeginJob, so remove it. --- src/qemu/qemu_domain.c | 41 ----------------------------------------- src/qemu/qemu_domain.h | 2 -- src/qemu/qemu_driver.c | 36 ++++++++++++++++++------------------ src/qemu/qemu_migration.c | 8 ++++---- 4 files changed, 22 insertions(+), 65 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index b9bf8a4..9a9088a 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -481,47 +481,6 @@ int qemuDomainObjBeginJob(virDomainObjPtr obj) return 0; } -/* - * This must be called by anything that will change the VM state - * in any way, or anything that will use the QEMU monitor. - */ -int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver ATTRIBUTE_UNUSED, - virDomainObjPtr obj) -{ - qemuDomainObjPrivatePtr priv = obj->privateData; - struct timeval now; - unsigned long long then; - - if (gettimeofday(&now, NULL) < 0) { - virReportSystemError(errno, "%s", - _("cannot get time of day")); - return -1; - } - then = timeval_to_ms(now) + QEMU_JOB_WAIT_TIME; - - virDomainObjLock(obj); - - while (priv->jobActive) { - if (virCondWaitUntil(&priv->jobCond, &obj->lock, then) < 0) { - int err = errno; - virDomainObjUnlock(obj); - if (err == ETIMEDOUT) - qemuReportError(VIR_ERR_OPERATION_TIMEOUT, - "%s", _("cannot acquire state change lock")); - else - virReportSystemError(errno, - "%s", _("cannot acquire job mutex")); - return -1; - } - } - priv->jobActive = QEMU_JOB_UNSPECIFIED; - priv->jobSignals = 0; - memset(&priv->jobSignalsData, 0, sizeof(priv->jobSignalsData)); - priv->jobStart = timeval_to_ms(now); - memset(&priv->jobInfo, 0, sizeof(priv->jobInfo)); - - return 0; -} /* * obj must be locked before calling, qemud_driver does not matter diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index 75ad5d9..500a882 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -95,8 +95,6 @@ void qemuDomainSetPrivateDataHooks(virCapsPtr caps); void qemuDomainSetNamespaceHooks(virCapsPtr caps); int qemuDomainObjBeginJob(virDomainObjPtr obj) ATTRIBUTE_RETURN_CHECK; -int qemuDomainObjBeginJobWithDriver(struct qemud_driver *driver, - virDomainObjPtr obj) ATTRIBUTE_RETURN_CHECK; void qemuDomainObjEndJob(virDomainObjPtr obj); void qemuDomainObjEnterMonitor(virDomainObjPtr obj); void qemuDomainObjExitMonitor(virDomainObjPtr obj); diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 887a865..e1b49e8 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -140,7 +140,7 @@ qemuAutostartDomain(void *payload, const void *name ATTRIBUTE_UNUSED, void *opaq virErrorPtr err; virResetLastError(); - if (qemuDomainObjBeginJobWithDriver(data->driver, vm) < 0) { + if (qemuDomainObjBeginJob(vm) < 0) { err = virGetLastError(); VIR_ERROR(_("Failed to start job on VM '%s': %s"), vm->def->name, @@ -1279,7 +1279,7 @@ static virDomainPtr qemudDomainCreate(virConnectPtr conn, const char *xml, def = NULL; - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) { + if (qemuDomainObjBeginJob(vm) < 0) { virDomainObjUnref(vm); goto cleanup; /* XXXX free the 'vm' we created ? */ } @@ -1351,7 +1351,7 @@ static int qemudDomainSuspend(virDomainPtr dom) { ret = 0; goto cleanup; } else { - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; if (!virDomainObjIsActive(vm)) { @@ -1401,7 +1401,7 @@ static int qemudDomainResume(virDomainPtr dom) { goto cleanup; } - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; if (!virDomainObjIsActive(vm)) { @@ -1493,7 +1493,7 @@ static int qemudDomainDestroy(virDomainPtr dom) { return -1; } - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; if (!virDomainObjIsActive(vm)) { @@ -1849,7 +1849,7 @@ static int qemudDomainSaveFlag(struct qemud_driver *driver, virDomainPtr dom, priv = vm->privateData; - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; priv->jobActive = QEMU_JOB_SAVE; @@ -2324,7 +2324,7 @@ static int qemudDomainCoreDump(virDomainPtr dom, goto cleanup; } - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; priv = vm->privateData; @@ -2416,7 +2416,7 @@ static void processWatchdogEvent(void *data, void *opaque) break; } - if (qemuDomainObjBeginJobWithDriver(driver, wdEvent->vm) < 0) + if (qemuDomainObjBeginJob(wdEvent->vm) < 0) break; if (!virDomainObjIsActive(wdEvent->vm)) { @@ -3162,7 +3162,7 @@ qemuDomainRestore(virConnectPtr conn, } def = NULL; - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; ret = qemuDomainSaveImageStartVM(conn, driver, vm, &fd, &header, path); @@ -3248,7 +3248,7 @@ static char *qemudDomainDumpXML(virDomainPtr dom, /* Don't delay if someone's using the monitor, just use * existing most recent data instead */ if (!priv->jobActive) { - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; qemuDomainObjEnterMonitor(vm); @@ -3469,7 +3469,7 @@ qemudDomainStartWithFlags(virDomainPtr dom, unsigned int flags) goto cleanup; } - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; if (virDomainObjIsActive(vm)) { @@ -3729,7 +3729,7 @@ static int qemudDomainAttachDevice(virDomainPtr dom, goto cleanup; } - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; if (!virDomainObjIsActive(vm)) { @@ -3909,7 +3909,7 @@ static int qemuDomainUpdateDeviceFlags(virDomainPtr dom, goto cleanup; } - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; if (!virDomainObjIsActive(vm)) { @@ -4013,7 +4013,7 @@ static int qemudDomainDetachDevice(virDomainPtr dom, goto cleanup; } - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; if (!virDomainObjIsActive(vm)) { @@ -6007,7 +6007,7 @@ qemuDomainSnapshotCreateActive(virConnectPtr conn, bool resume = false; int ret = -1; - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) return -1; if (vm->state == VIR_DOMAIN_RUNNING) { @@ -6351,7 +6351,7 @@ static int qemuDomainRevertToSnapshot(virDomainSnapshotPtr snapshot, vm->current_snapshot = snap; - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; if (snap->def->state == VIR_DOMAIN_RUNNING @@ -6617,7 +6617,7 @@ static int qemuDomainSnapshotDelete(virDomainSnapshotPtr snapshot, goto cleanup; } - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; if (flags & VIR_DOMAIN_SNAPSHOT_DELETE_CHILDREN) { @@ -6687,7 +6687,7 @@ static int qemuDomainMonitorCommand(virDomainPtr domain, const char *cmd, hmp = !!(flags & VIR_DOMAIN_QEMU_MONITOR_COMMAND_HMP); - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; qemuDomainObjEnterMonitor(vm); ret = qemuMonitorArbitraryCommand(priv->mon, cmd, result, hmp); diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 5076cd5..fb9009f 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -291,7 +291,7 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver, def = NULL; priv = vm->privateData; - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; priv->jobActive = QEMU_JOB_MIGRATION_OUT; @@ -508,7 +508,7 @@ qemuMigrationPrepareDirect(struct qemud_driver *driver, def = NULL; priv = vm->privateData; - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; priv->jobActive = QEMU_JOB_MIGRATION_OUT; @@ -1047,7 +1047,7 @@ int qemuMigrationPerform(struct qemud_driver *driver, int resume = 0; qemuDomainObjPrivatePtr priv = vm->privateData; - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; priv->jobActive = QEMU_JOB_MIGRATION_OUT; @@ -1181,7 +1181,7 @@ qemuMigrationFinish(struct qemud_driver *driver, priv->jobActive = QEMU_JOB_NONE; memset(&priv->jobInfo, 0, sizeof(priv->jobInfo)); - if (qemuDomainObjBeginJobWithDriver(driver, vm) < 0) + if (qemuDomainObjBeginJob(vm) < 0) goto cleanup; /* Did the migration go as planned? If yes, return the domain -- 1.7.3.1 -- Thanks, Hu Tao

On 04/06/2011 01:20 AM, Hu Tao wrote:
The body of qemuDomainObjBeginJobWithDriver is the same as qemuDomainObjBeginJob, so remove it. --- src/qemu/qemu_domain.c | 41 ----------------------------------------- src/qemu/qemu_domain.h | 2 -- src/qemu/qemu_driver.c | 36 ++++++++++++++++++------------------ src/qemu/qemu_migration.c | 8 ++++---- 4 files changed, 22 insertions(+), 65 deletions(-)
You also need to patch src/qemu/THREADS.txt to reflect this consolidation. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

On 04/06/2011 01:18 AM, Hu Tao wrote:
This series adds a virObject structure that manages reference-counting.
Some notes about referece-counting introduced by this series:
A thread owns a virObject by incrementing its reference-count by 1. If a thread owns a virObject, the virObject is guarenteed not be freed until the thread releases ownership by decrementing its reference-count by 1. A thread can't access a virObject after it releases the ownership of virObject because it can be freed at anytime.
A thread can own a virObject legally in these ways:
- a thread owns a virObject that it creates. - a thread owns a virObject if another thread passes the ownership to it. Example: qemuMonitorOpen - a thread gets a virObject from a container. Example: virDomainFindByUUID - a container owns a virObject by incrementing its reference-count by 1 before adding it to the container - if a virObject is removed from a container its reference-count must be decremented by 1
By following these rules, there is no need to protect operations on an object's reference-count by an external lock. (like in old ways virDomainObj lock protects qemu monitor's ref-count.)
These rules _need_ to be part of the patch series, preferably added at the same time as patch 1/6 (see for example how src/qemu/THREADS.txt documents threading issues related to qemu). I don't know whether virobject.h or a standalone document is better (but if you do a standalone document, then virobject.h should at least mention it). -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

On Wed, Apr 06, 2011 at 03:30:50PM -0600, Eric Blake wrote:
On 04/06/2011 01:18 AM, Hu Tao wrote:
This series adds a virObject structure that manages reference-counting.
Some notes about referece-counting introduced by this series:
A thread owns a virObject by incrementing its reference-count by 1. If a thread owns a virObject, the virObject is guarenteed not be freed until the thread releases ownership by decrementing its reference-count by 1. A thread can't access a virObject after it releases the ownership of virObject because it can be freed at anytime.
A thread can own a virObject legally in these ways:
- a thread owns a virObject that it creates. - a thread owns a virObject if another thread passes the ownership to it. Example: qemuMonitorOpen - a thread gets a virObject from a container. Example: virDomainFindByUUID - a container owns a virObject by incrementing its reference-count by 1 before adding it to the container - if a virObject is removed from a container its reference-count must be decremented by 1
By following these rules, there is no need to protect operations on an object's reference-count by an external lock. (like in old ways virDomainObj lock protects qemu monitor's ref-count.)
These rules _need_ to be part of the patch series, preferably added at the same time as patch 1/6 (see for example how src/qemu/THREADS.txt documents threading issues related to qemu). I don't know whether virobject.h or a standalone document is better (but if you do a standalone document, then virobject.h should at least mention it).
OK. Will put it in virobject.h in next version.
-- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org
participants (4)
-
Daniel P. Berrange
-
Eric Blake
-
Hu Tao
-
Wen Congyang