September 2012 - Devel - libvirt List Archives

[libvirt] [PATCH 00/12] Structured syslog (Lumberjack/CEE) support
by Miloslav Trmač 09 Oct '12

09 Oct '12

This patch set adds structured syslog support to libvirt. In short, the idea is to embed JSON in a syslog record. This format is used in a new log output type called "syslog/json". Example "syslog/json" record (line-wrapped for readability): Sep 20 19:59:40 kulicka libvirt: @cee: {"msg":"libvirt version: 0.10.1", "category":"../../src/test/test_driver.c","priority":"info", "funcname":"virLogVMessage","line":769, "timestamp":"2012-09-20 17:59:40.853+0000"} compared with existing "syslog" record format: Sep 20 20:02:24 kulicka libvirt: 5896: info : libvirt version: 0.10.1 This JSON embedding format is defined by the Lumberjack project led by the major Open Source syslog implementations (https://fedorahosted.org/lumberjack/) The format is also a base for a future MITRE CEE standard. Some of the fields are specified at https://fedorahosted.org/lumberjack/wiki/FieldList , but applications are free to use whatever fields make sense for them. The JSON structure makes easy to process logs by programs without writing code that understands the specific format used by each application, or using unreliable regexps (https://fedorahosted.org/ceelog/ is a companion project that allows processing such logs). Current systems will store and these records unmodified and present them to users unmodified, but tools (and probably /var/log/messages) will be able to show only the "msg" field; this allows us to add more detailed information without worsening the user experience (the example above already shows added information that is currently being thrown away when logging to syslog). The patch set, aside from building the required infrastructure, has two main components: * Add the "syslog/json" output type, and use it to record all information passed to virLogMessage. The example result is shown above. This primarily makes it easier to parse the message. * Allow callers of virLogMessage to add arbitrary additional JSON fields, and use it in virRaiseErrorFull and its callers. Because the additional fields for errors include virErrorDomain and virErrorNumber, this allows _programs_ to _automatically_ detect the event and react to it, perhaps alerting the user about a situation that requires immediate attention, or automatically suggesting a solution. This has not been possible until now because the current logs contain only a translated error message, so the log record doesn't contain anything stable that could be detected by a program. It is assumed that other parts of libvirt may add more event-specific JSON fields in the future, based on demand by applications that process the logs; I have, however, no immediate plans to add more. Miloslav Trmač (12): Drop unused return value of virLogOutputFunc Add JSON API for preallocated objects. Split yajl_gen allocation into a separate function. Split adding object properties from virJSONValueToStringOne Add a JSON generator API. Pass the "raw" log message to each virLogOutputFunc Split priority conversion from virLogOutputToSyslog Add "syslog/json" log output format. Add a JSON properties parameter to virLog{,V}Message Add a JSON properties parameter to virLogOutputFunc Add detailed error information to JSON syslog Add a test for the new error reporting mechanism. docs/logging.html.in | 3 + src/libvirt_private.syms | 4 + src/uml/uml_conf.c | 2 +- src/util/json.c | 232 +++++++++++++++++++++++++++++++++++++++++----- src/util/json.h | 26 ++++++ src/util/logging.c | 233 ++++++++++++++++++++++++++++++++++------------- src/util/logging.h | 35 +++---- src/util/viraudit.c | 4 +- src/util/virterror.c | 32 ++++++- tests/Makefile.am | 6 +- tests/errorjsontest.c | 211 ++++++++++++++++++++++++++++++++++++++++++ tests/testutils.c | 7 +- 12 files changed, 684 insertions(+), 111 deletions(-) create mode 100644 tests/errorjsontest.c -- 1.7.11.4

4 24

[libvirt] [PATCH 0/3] non-shared migration without target disk images
by liguang 09 Oct '12

09 Oct '12

by now, if you want a non-shared migration, you have to create same files at target as source side, which seems intorlerable! this issue was reported by Reinier Schoof http://www.redhat.com/archives/libvir-list/2011-December/msg00451.html These patches try to let migrate --copy-storage-* work without pre-exist disk image files at target side. we may also hope qemu to aware of this issue hooks/qemu | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/Makefile.am | 3 ++ src/util/hooks.c | 2 +- 3 files changed, 59 insertions(+), 1 deletions(-)

3 7

[libvirt] [libvirt-list] question about the commands: domhostname and hostname
by Li Zhang 09 Oct '12

09 Oct '12

Hi all, I am testing the virsh commands, and I found that these two commands: * domhostname * hostname >From the manual of virsh commands: * hostname Print the hypervisor hostname. * domhostname domain Returns the hostname of a domain, if the hypervisor makes it available It seems that both of them want to get the host name. What's the difference between them? >From the source code, domhostname only is supported for openvz on 0.10.1. Will this be only for openvz in the future? Any idea? Thanks a lot in advance. -:) -- Best Regards -Li

3 7

[libvirt] [PATCH] fix-up an implicit build dead-loop in cfg.mk
by liguang 09 Oct '12

09 Oct '12

if gnulib submodule happened to be dirty, build process will fall into '_autogen' target trap in cfg.mk recursively, so break this dead-loop. Signed-off-by: liguang <lig.fnst(a)cn.fujitsu.com> --- cfg.mk | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/cfg.mk b/cfg.mk index bbfd4a2..92966d5 100644 --- a/cfg.mk +++ b/cfg.mk @@ -699,7 +699,7 @@ ifeq (0,$(MAKELEVEL)) test "$$stamp" = "$$actual"; echo $$?) _clean_requested = $(filter %clean,$(MAKECMDGOALS)) ifeq (1,$(_update_required)$(_clean_requested)) - $(info INFO: gnulib update required; running ./autogen.sh first) + $(error ERR: gnulib update required; running ./autogen.sh first) Makefile: _autogen endif endif -- 1.7.2.5

3 3

[libvirt] Core dump caused by misusing openssl in multithread scenario!
by Benjamin Wang (gendwang) 08 Oct '12

08 Oct '12

Hi, I am running libvirt with ESXi driver in multithread scenario to access ESXi by https. Sometimes a core dump will be generated as following: #0 0x0000003f9b030265 in raise () from /lib64/libc.so.6 #1 0x0000003f9b031d10 in abort () from /lib64/libc.so.6 #2 0x0000003f9b06a84b in __libc_message () from /lib64/libc.so.6 #3 0x0000003f9b072fae in _int_malloc () from /lib64/libc.so.6 #4 0x0000003f9b074cde in malloc () from /lib64/libc.so.6 #5 0x0000003f9b07963b in strerror () from /lib64/libc.so.6 #6 0x0000003fa188032a in ERR_load_ERR_strings () from /lib64/libcrypto.so.6 #7 0x0000003fa187fde9 in ERR_load_crypto_strings () from /lib64/libcrypto.so.6 #8 0x0000003fa48309d9 in SSL_load_error_strings () from /lib64/libssl.so.6 #9 0x00002aaaba8e612e in Curl_ossl_init () from /opt/CSCOppm-unit/hypervisor/libcurl/lib/libcurl.so.4 #10 0x00002aaaba8ee6c1 in curl_global_init () from /opt/CSCOppm-unit/hypervisor/libcurl/lib/libcurl.so.4 #11 0x00002aaaba8ee6f8 in curl_easy_init () from /opt/CSCOppm-unit/hypervisor/libcurl/lib/libcurl.so.4 #12 0x00002aaaba0d932b in esxVI_SessionIsActive (ctx=0x2aaac093ca80, sessionID=0x2aaac06932a0 "`3i\300\252*", userName=0x2aaac0ae6e80 "root", output=0xffffffffffffffff) at esx/esx_vi_methods.generated.c:599 #13 0x00002aaaba0c7a60 in esxStorageVolumeLookupByKey (conn=0x7412, key=0x76c1 <Address 0x76c1 out of bounds>) at esx/esx_storage_driver.c:825 I checked that currently ESXi driver didn't initialize openssl. Because libcurl will not handle openssl for multi-thread. According to openssl API, libvirt should register two methods to support mutli-threads. The detailed description is as following: http://www.openssl.org/docs/crypto/threads.html I have changed code as following: 1. virInitialize() in libvirt.c Old Code: int virInitialize(void) { ... virLogSetFromEnv(); virNetTLSInit(); ... } New Code: int virInitialize(void) { ... virLogSetFromEnv(); virNetTLSInit(); virOpenSSLInit(); ... } 2. In virnetServer.c New Code: pthread_mutex_t *lock_cs; long *lock_count; void virOpenSSLLockCallback(int mode, int type, const char *file ATTRIBUTE_UNUSED, int line ATTRIBUTE_UNUSED) { if (mode & CRYPTO_LOCK) { pthread_mutex_lock(&(lock_cs[type])); lock_count[type]++; } else { pthread_mutex_unlock(&(lock_cs[type])); } } unsigned long virOpenSSLIdCallback(void) { unsigned long ret; ret=(unsigned long)pthread_self(); return(ret); } void virOpenSSLInit(void) { int i; lock_cs=OPENSSL_malloc(CRYPTO_num_locks() * sizeof(pthread_mutex_t)); lock_count=OPENSSL_malloc(CRYPTO_num_locks() * sizeof(long)); for (i=0; i<CRYPTO_num_locks(); i++) { lock_count[i]=0; pthread_mutex_init(&(lock_cs[i]),NULL); } CRYPTO_set_id_callback(virOpenSSLIdCallback); CRYPTO_set_locking_callback(virOpenSSLLockCallback); } To be honest, virOpenSSLInit/ virOpenSSLIdCallback/ virOpenSSLLockCallback should not be defined in this file. But It seems that Makefile generated by autoconfig can't handle the new file recursively. What about this solution? If you have any comments, please feel free to contact me. BTW: If I add a new source/header file, is there a simple way to change Makefile? B.R. Benjamin Wang

3 10

[libvirt] [PATCH v2 0/2] python: cleanup vcpu related binding APIs
by Guannan Ren 08 Oct '12

08 Oct '12

The first version https://www.redhat.com/archives/libvir-list/2012-April/msg00848.html Sorry for so late. I will go on adding error handling and make it strong for python binding APIs if necessary Guannan Ren python: cleanup vcpu related binding APIs python: return error if PyObject obj is NULL for unwrapper helper functions python/libvirt-override.c | 171 ++++++++++++++++++++++++++++++++++++++-------------- python/typewrappers.c | 43 +++++++++++++- 2 files changed, 164 insertions(+), 50 deletions(-)

2 9

[libvirt] [PATCH 0/7] Add support for lock failure action
by Jiri Denemark 08 Oct '12

08 Oct '12

When sanlock is used as a locking driver and sanlock deamon loses access to its lockspace, it automatically kills off the domains that had their locks stored there. Apparently some management apps would like to change this behavior by configuring what should happen when locks are lost. Everything except the sanlock_helper code from patch 7/7 should be stable. The helper code may need some additional work (perhaps even touching other parts of libvirtd internals) because some lock failure action does not work as expected. For example, when automatic disk locks are enabled, the "pause" action will result in libvirtd calling back to sanlock and since it doesn't have access to the lockspace, the call would just hang. However, since currently the only requestor for this feature is VDSM and it doesn't use automatic disk locks, the sanlock_helper could work for them without any modifications. And since current sanlock_helper implementation is supposed to be changed when libvirtd gains administrative interface (see patch 7/7 for more details) we may even just live with its limitations for the time being. Jiri Denemark (7): conf: Generalize life cycle actions to event actions conf: Add on_lockfailure event configuration qemu: Use macro instead of "qemu" in the context of URI scheme locking: Add const char * parameter to avoid ugly typecasts locking: Pass hypervisor driver name when acquiring locks locking: Add support for lock failure action locking: Implement lock failure action in sanlock driver docs/formatdomain.html.in | 37 +++++-- docs/internals/locking.html.in | 8 ++ docs/schemas/domaincommon.rng | 30 +++++- libvirt.spec.in | 1 + src/Makefile.am | 13 ++- src/conf/domain_conf.c | 85 +++++++++------ src/conf/domain_conf.h | 18 +++- src/libvirt_private.syms | 2 + src/locking/domain_lock.c | 36 ++++--- src/locking/domain_lock.h | 4 + src/locking/lock_driver.h | 8 +- src/locking/lock_driver_nop.c | 1 + src/locking/lock_driver_sanlock.c | 114 +++++++++++++++++--- src/locking/lock_manager.c | 10 +- src/locking/lock_manager.h | 1 + src/locking/sanlock_helper.c | 214 ++++++++++++++++++++++++++++++++++++++ src/qemu/qemu_conf.h | 2 + src/qemu/qemu_driver.c | 21 ++-- src/qemu/qemu_hotplug.c | 15 ++- src/qemu/qemu_process.c | 4 +- 20 files changed, 537 insertions(+), 87 deletions(-) create mode 100644 src/locking/sanlock_helper.c -- 1.7.12

2 14

[libvirt] [PATCH] python: keep consistent handling of Python integer conversion
by Guannan Ren 08 Oct '12

08 Oct '12

libvirt_ulonglongUnwrap requires the integer type of python obj. But libvirt_longlongUnwrap still could handle python obj of Pyfloat_type which causes the float value to be rounded up to an integer. For example >>> dom.setSchedulerParameters({'vcpu_quota': 0.88}) 0 libvirt_longlongUnwrap treats 0.88 as a valid value 0 However >>> dom.setSchedulerParameters({'cpu_shares': 1000.22}) libvirt_ulonglongUnwrap will throw out an error "TypeError: an integer is required" The patch make this consistent. --- python/typewrappers.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/python/typewrappers.c b/python/typewrappers.c index 7580689..d633603 100644 --- a/python/typewrappers.c +++ b/python/typewrappers.c @@ -220,7 +220,7 @@ libvirt_ulongUnwrap(PyObject *obj, unsigned long *val) int libvirt_longlongUnwrap(PyObject *obj, long long *val) { - long long llong_val; + long long llong_val = -1; if (!obj) { PyErr_SetString(PyExc_TypeError, "unexpected type"); @@ -230,7 +230,11 @@ libvirt_longlongUnwrap(PyObject *obj, long long *val) /* If obj is of PyInt_Type, PyLong_AsLongLong * will call PyInt_AsLong() to handle it automatically. */ - llong_val = PyLong_AsLongLong(obj); + if (PyInt_Check(obj) || PyLong_Check(obj)) + llong_val = PyLong_AsLongLong(obj); + else + PyErr_SetString(PyExc_TypeError, "an integer is required"); + if ((llong_val == -1) && PyErr_Occurred()) return -1; -- 1.7.11.2

2 2

[libvirt] [PATCH v3] esx: Support VLAN tags in virtual network port groups
by Matthias Bolte 07 Oct '12

07 Oct '12

--- v2: Use network level VLAN config if there is no portgroup specific VLAN config given. v3: Add ESX_VLAN_TRUNK define for magic number 4095 src/esx/esx_network_driver.c | 70 +++++++++++++++++++++++++++++++++++++++--- 1 files changed, 65 insertions(+), 5 deletions(-) diff --git a/src/esx/esx_network_driver.c b/src/esx/esx_network_driver.c index 09d46d3..21eabbe 100644 --- a/src/esx/esx_network_driver.c +++ b/src/esx/esx_network_driver.c @@ -45,6 +45,9 @@ */ verify(MD5_DIGEST_SIZE == VIR_UUID_BUFLEN); +/* ESX utilizes the VLAN ID of 4095 to mean that this port is in trunk mode */ +#define ESX_VLAN_TRUNK 4095 + static virDrvOpenStatus @@ -489,7 +492,42 @@ esxNetworkDefineXML(virConnectPtr conn, const char *xml) goto cleanup; } - hostPortGroupSpec->vlanId->value = 0; + if (def->portGroups[i].vlan.trunk) { + /* FIXME: Change this once tag-less trunk-mode is supported */ + if (def->portGroups[i].vlan.nTags != 1 || + *def->portGroups[i].vlan.tag != ESX_VLAN_TRUNK) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("VLAN tag has to be %d in trunk mode"), + ESX_VLAN_TRUNK); + goto cleanup; + } + + hostPortGroupSpec->vlanId->value = ESX_VLAN_TRUNK; + } else if (def->portGroups[i].vlan.nTags == 1) { + hostPortGroupSpec->vlanId->value = *def->portGroups[i].vlan.tag; + } else if (def->portGroups[i].vlan.nTags > 1) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("Can apply one VLAN tag per port group only")); + goto cleanup; + } else if (def->vlan.trunk) { + /* FIXME: Change this once tag-less trunk-mode is supported */ + if (def->vlan.nTags != 1 || *def->vlan.tag != ESX_VLAN_TRUNK) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("VLAN tag has to be %d in trunk mode"), + ESX_VLAN_TRUNK); + goto cleanup; + } + + hostPortGroupSpec->vlanId->value = ESX_VLAN_TRUNK; + } else if (def->vlan.nTags == 1) { + hostPortGroupSpec->vlanId->value = *def->vlan.tag; + } else if (def->vlan.nTags > 1) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("Can apply one VLAN tag per port group only")); + goto cleanup; + } else { + hostPortGroupSpec->vlanId->value = 0; + } if (def->portGroups[i].bandwidth != NULL) { if (esxBandwidthToShapingPolicy @@ -519,6 +557,8 @@ esxNetworkDefineXML(virConnectPtr conn, const char *xml) network = virGetNetwork(conn, hostVirtualSwitch->name, md5); cleanup: + /* FIXME: need to remove virtual switch if adding port groups failed */ + virNetworkDefFree(def); esxVI_HostVirtualSwitch_Free(&hostVirtualSwitch); esxVI_HostPortGroup_Free(&hostPortGroupList); @@ -695,6 +735,7 @@ esxNetworkGetXMLDesc(virNetworkPtr network_, unsigned int flags) esxVI_String *hostPortGroupKey = NULL; esxVI_String *networkName = NULL; virNetworkDefPtr def; + virPortGroupDefPtr portGroup; if (esxVI_EnsureSession(priv->primary) < 0) { return NULL; @@ -824,9 +865,12 @@ esxNetworkGetXMLDesc(virNetworkPtr network_, unsigned int flags) for (networkName = networkNameList; networkName != NULL; networkName = networkName->_next) { if (STREQ(networkName->value, hostPortGroup->spec->name)) { - def->portGroups[def->nPortGroups].name = strdup(networkName->value); + portGroup = &def->portGroups[def->nPortGroups]; + ++def->nPortGroups; + + portGroup->name = strdup(networkName->value); - if (def->portGroups[def->nPortGroups].name == NULL) { + if (portGroup->name == NULL) { virReportOOMError(); goto cleanup; } @@ -834,13 +878,29 @@ esxNetworkGetXMLDesc(virNetworkPtr network_, unsigned int flags) if (hostPortGroup->spec->policy != NULL) { if (esxShapingPolicyToBandwidth (hostPortGroup->spec->policy->shapingPolicy, - &def->portGroups[def->nPortGroups].bandwidth) < 0) { + &portGroup->bandwidth) < 0) { ++def->nPortGroups; goto cleanup; } } - ++def->nPortGroups; + if (hostPortGroup->spec->vlanId->value > 0) { + if (hostPortGroup->spec->vlanId->value == ESX_VLAN_TRUNK) { + portGroup->vlan.trunk = true; + } + + /* FIXME: Remove this once tag-less trunk-mode is supported */ + portGroup->vlan.nTags = 1; + + if (VIR_ALLOC_N(portGroup->vlan.tag, 1) < 0) { + virReportOOMError(); + goto cleanup; + } + + *portGroup->vlan.tag = + hostPortGroup->spec->vlanId->value; + } + break; } } -- 1.7.4.1

2 3

[libvirt] Two core dumps are generated in multi-thread scenarios
by Benjamin Wang (gendwang) 07 Oct '12

07 Oct '12

Hi, I found two core dumps generated in multi-thread scenarios in ESX part. Case1: libcurl support multi-thread core dump: #12 0x00002aaabea89712 in addbyter () from /usr/local/lib/libcurl.so.4 #13 0x00002aaabea89b86 in dprintf_formatf () from /usr/local/lib/libcurl.so.4 #14 0x00002aaabea8b055 in curl_mvsnprintf () from /usr/local/lib/libcurl.so.4 #15 0x00002aaabea7678f in Curl_failf () from /usr/local/lib/libcurl.so.4 #16 0x00002aaabea6d871 in Curl_resolv_timeout () from /usr/local/lib/libcurl.so.4 #17 0x00000006e8a8f230 in ?? () Fix code: esxVI_CURL_Connect() in esx_vi.c: I add a new line as following: curl_easy_setopt(curl->handle, CURLOPT_NOSIGNAL, 1); Case2: libssl support multi-thread core dump: #0 0x0000003f9b030265 in raise () from /lib64/libc.so.6 #1 0x0000003f9b031d10 in abort () from /lib64/libc.so.6 #2 0x0000003f9b06a84b in __libc_message () from /lib64/libc.so.6 #3 0x0000003f9b072fae in _int_malloc () from /lib64/libc.so.6 #4 0x0000003f9b074cde in malloc () from /lib64/libc.so.6 #5 0x0000003f9b07963b in strerror () from /lib64/libc.so.6 #6 0x0000003fa188032a in ERR_load_ERR_strings () from /lib64/libcrypto.so.6 #7 0x0000003fa187fde9 in ERR_load_crypto_strings () from /lib64/libcrypto.so.6 #8 0x0000003fa48309d9 in SSL_load_error_strings () from /lib64/libssl.so.6 #9 0x00002aaaba8e612e in Curl_ossl_init () from /opt/CSCOppm-unit/hypervisor/libcurl/lib/libcurl.so.4 #10 0x00002aaaba8ee6c1 in curl_global_init () from /opt/CSCOppm-unit/hypervisor/libcurl/lib/libcurl.so.4 #11 0x00002aaaba8ee6f8 in curl_easy_init () from /opt/CSCOppm-unit/hypervisor/libcurl/lib/libcurl.so.4 #12 0x00002aaaba0d932b in esxVI_RegisterVM_Task (ctx=0x2aaaba0d96d1, _this=0x5cf54b20, path=0x50e921c0 "10.74.125.50", name=0x2aaac0ae6e80 "root", asTemplate=3228119712, pool=0x5cf54b20, host=0x2aaac0693270, output=0x50e921a0) at esx/esx_vi_methods.generated.c:480 Possible Problem: Two callback functions(locking_function and threadid_func) need to be set. http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION Would you help to give some comments about this two core dump? B.R. Benjamin Wang

3 5