April 2011 - Devel - libvirt List Archives

[libvirt] [PATCH 0/6] Add disk streaming API to libvirt
by Adam Litke 12 May '11

12 May '11

I've been working with Anthony Liguori and Stefan Hajnoczi to enable data streaming to copy-on-read disk images in qemu. This work is working its way through review and I expect it to be upstream soon as part of the support for the new QED disk image format. Disk streaming is extremely useful when provisioning domains from a central repository of template images. Currently the domain must be provisioned by either: 1) copying the template image to local storage before the VM can be started or, 2) creating a qcow2 image that backs to a base image in the remote repository. Option 1 can introduce a significant delay when provisioning large disks. Option 2 introduces a permanent dependency on a remote service and increased network load to satisfy disk reads. Device streaming provides the "instant-on" benefits of option 2 without introducing a permanent dependency to the image repository. Once the VM is started, the contents of the disk can be streamed to the local image in parallel. Once streaming is finished, the domain has a complete and coherent copy of the image and no longer depends on the central image repository. Qemu will support two streaming modes: full device and single sector. Full device streaming is the easiest to use because one command will cause the whole device to be streamed as fast as possible. Single sector mode can be used if one wants to throttle streaming to reduce I/O pressure. In this mode, a management tool issues individual commands to stream single sectors. To enable this support in libvirt, I propose the following API... virDomainStreamDisk() will start or stop a full device stream or stream a single sector of a device. The behavior is controlled by setting virDomainStreamDiskFlags. When either starting or stopping a full device stream, the return value is either 0 or -1 to indicate whether the operation succeeded. For a single sector stream, a device offset is returned (or -1 on failure). This value can be used to continue streaming with a subsequent call to virDomainStreamDisk(). virDomainStreamDiskInfo() returns information about active full device streams (the device alias, current streaming position, and total size). Adam Litke (6): Add new API virDomainStreamDisk[Info] to header and drivers virDomainStreamDisk: Add public symbols to libvirt API Implement disk streaming in the qemu driver Add disk streaming support to the remote driver Add new disk streaming commands to virsh python: Add python bindings for virDomainStreamDisk[Info] b/daemon/remote.c | 96 ++++++++++++++++++++ b/daemon/remote_dispatch_args.h | 2 b/daemon/remote_dispatch_prototypes.h | 16 +++ b/daemon/remote_dispatch_ret.h | 2 b/daemon/remote_dispatch_table.h | 10 ++ b/include/libvirt/libvirt.h.in | 34 +++++++ b/python/generator.py | 4 b/python/libvirt-override-api.xml | 5 + b/python/libvirt-override.c | 46 +++++++++ b/src/driver.h | 11 ++ b/src/esx/esx_driver.c | 2 b/src/libvirt.c | 115 ++++++++++++++++++++++++ b/src/libvirt_public.syms | 5 + b/src/lxc/lxc_driver.c | 2 b/src/openvz/openvz_driver.c | 2 b/src/phyp/phyp_driver.c | 2 b/src/qemu/qemu_driver.c | 77 +++++++++++++++- b/src/qemu/qemu_monitor.c | 42 ++++++++ b/src/qemu/qemu_monitor.h | 6 + b/src/qemu/qemu_monitor_json.c | 108 ++++++++++++++++++++++ b/src/qemu/qemu_monitor_json.h | 7 + b/src/qemu/qemu_monitor_text.c | 162 ++++++++++++++++++++++++++++++++++ b/src/qemu/qemu_monitor_text.h | 8 + b/src/remote/remote_driver.c | 87 +++++++++++++++++- b/src/remote/remote_protocol.c | 63 +++++++++++++ b/src/remote/remote_protocol.h | 51 ++++++++++ b/src/remote/remote_protocol.x | 37 +++++++ b/src/test/test_driver.c | 2 b/src/uml/uml_driver.c | 2 b/src/vbox/vbox_tmpl.c | 2 b/src/vmware/vmware_driver.c | 2 b/src/xen/xen_driver.c | 2 b/tools/virsh.c | 134 +++++++++++++++++++++++++++- python/generator.py | 3 src/qemu/qemu_driver.c | 2 src/remote/remote_driver.c | 2 36 files changed, 1144 insertions(+), 9 deletions(-)

7 26

[libvirt] [PATCH] build: Fix problem of building Python bindings
by Osier Yang 10 May '11

10 May '11

If one specify "--with-python=yes" but no python-devel package is installed, we ignore it with just a notice message, which doesn't give clear guide to user. --- configure.ac | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diff --git a/configure.ac b/configure.ac index 190bf40..758c893 100644 --- a/configure.ac +++ b/configure.ac @@ -1917,15 +1917,13 @@ if test "$with_python" != "no" ; then then PYTHON_INCLUDES=-I/usr/include/python$PYTHON_VERSION else - AC_MSG_NOTICE([Could not find python$PYTHON_VERSION/Python.h, disabling bindings]) - with_python=no + AC_MSG_ERROR([You must install python-devel to build Python bindings]) fi fi fi fi else - AC_MSG_NOTICE([Could not find python interpreter, disabling bindings]) - with_python=no + AC_MSG_ERROR([You must install python to build Python bindings]) fi else AC_MSG_NOTICE([Could not find python in $with_python, disabling bindings]) -- 1.7.4

3 4

[libvirt] [PATCH 00/22] Extend remote generator to generate function bodies too
by Matthias Bolte 07 May '11

07 May '11

Richard W.M. Jones suggested [1] that the code that directly deals with the XDR protocol should be generated. The remote_generate_stubs.pl script already generates all the headers, just the bodies in the daemon and remote driver are manually written. But most of the functions just follow simple patterns. So I extended the generator to exploit this patterns and move 11 kLOC code from manually written to generated code. During this I came a cross many small variations and problems in the XDR protocol. For example, NWFilterDefineXML has a flags parameter in the public API, but it's not transferred in the XDR protocol. Another things is the variations in the usage of unsigned VS signed types. This comes in two forms. public API VS XDR procotol and in between different functions. For example, some functions use int for the flags paramater and some use unsigned int. This results in quite a lot of special case handling in the generator. cfg.mk | 10 +- daemon/Makefile.am | 46 +- daemon/qemu_dispatch_args.h | 2 +- daemon/qemu_dispatch_bodies.c | 6 + daemon/qemu_dispatch_prototypes.h | 2 +- daemon/qemu_dispatch_ret.h | 2 +- daemon/qemu_dispatch_table.h | 2 +- daemon/remote.c | 5765 +---------------------------------- daemon/remote_dispatch_args.h | 2 +- daemon/remote_dispatch_bodies.c | 5933 +++++++++++++++++++++++++++++++++++ daemon/remote_dispatch_prototypes.h | 80 +- daemon/remote_dispatch_ret.h | 2 +- daemon/remote_dispatch_table.h | 158 +- daemon/remote_generate_stubs.pl | 195 -- daemon/remote_generator.pl | 1198 +++++++ po/POTFILES.in | 1 + src/Makefile.am | 13 +- src/remote/qemu_client_bodies.c | 4 + src/remote/qemu_protocol.c | 2 +- src/remote/qemu_protocol.h | 2 +- src/remote/qemu_protocol.x | 2 +- src/remote/remote_client_bodies.c | 4664 +++++++++++++++++++++++++++ src/remote/remote_driver.c | 4907 +---------------------------- src/remote/remote_protocol.c | 26 +- src/remote/remote_protocol.h | 26 +- src/remote/remote_protocol.x | 34 +- src/remote_protocol-structs | 26 +- 27 files changed, 12093 insertions(+), 11017 deletions(-) [1] https://www.redhat.com/archives/libvir-list/2011-April/msg00884.html Matthias

4 31

[libvirt] [PATCH] virsh: fix regression in log to file
by Supriya Kannery 06 May '11

06 May '11

Commit 36deff04 introduced a regression due to which virsh is not able to log to a file - msg_buf was changed from an array to a pointer without corresponding change to usage of "sizeof()". Fix regression in virsh logging Signed-off-by: Supriya Kannery <supriyak(a)in.ibm.com> --- tools/virsh.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) Index: libvirt/tools/virsh.c =================================================================== --- libvirt.orig/tools/virsh.c +++ libvirt/tools/virsh.c @@ -12257,7 +12257,7 @@ vshOutputLogFile(vshControl *ctl, int lo */ gettimeofday(&stTimeval, NULL); stTm = localtime(&stTimeval.tv_sec); - snprintf(msg_buf, sizeof(msg_buf), + snprintf(msg_buf, MSG_BUFFER, "[%d.%02d.%02d %02d:%02d:%02d ", (1900 + stTm->tm_year), (1 + stTm->tm_mon), @@ -12265,7 +12265,7 @@ vshOutputLogFile(vshControl *ctl, int lo (stTm->tm_hour), (stTm->tm_min), (stTm->tm_sec)); - snprintf(msg_buf + strlen(msg_buf), sizeof(msg_buf) - strlen(msg_buf), + snprintf(msg_buf + strlen(msg_buf), MSG_BUFFER - strlen(msg_buf), "%s] ", SIGN_NAME); switch (log_level) { case VSH_ERR_DEBUG: @@ -12287,13 +12287,13 @@ vshOutputLogFile(vshControl *ctl, int lo lvl = LVL_DEBUG; break; } - snprintf(msg_buf + strlen(msg_buf), sizeof(msg_buf) - strlen(msg_buf), + snprintf(msg_buf + strlen(msg_buf), MSG_BUFFER - strlen(msg_buf), "%s ", lvl); - vsnprintf(msg_buf + strlen(msg_buf), sizeof(msg_buf) - strlen(msg_buf), + vsnprintf(msg_buf + strlen(msg_buf), MSG_BUFFER - strlen(msg_buf), msg_format, ap); if (msg_buf[strlen(msg_buf) - 1] != '\n') - snprintf(msg_buf + strlen(msg_buf), sizeof(msg_buf) - strlen(msg_buf), "\n"); + snprintf(msg_buf + strlen(msg_buf), MSG_BUFFER - strlen(msg_buf), "\n"); /* write log */ if (safewrite(ctl->log_fd, msg_buf, strlen(msg_buf)) < 0) {

3 9

[libvirt] [PATCH 0/6] Introduce a new migration protocol to QEMU driver
by Daniel P. Berrange 06 May '11

06 May '11

The current migration protocol has several flaws - No initial hook on the source host to do work before the dst VM is launched - No ability to restart src VM if dst fails to recv all migration data, but src successfully sent it all This introduces a new 5 step migration process to address this limitation. To support features such as seemless migration of SPICE clients, and lock driver state passing this now makes use of the migration cookie feature too

4 31

[libvirt] [PATCH] lxc: Do not try to reconnect inactive domain when do lxcStartup
by Osier Yang 04 May '11

04 May '11

Otherwise if there are inactive lxc domains, lxcStartup will try to reconnect to sockets of these domains, which results in errors in libvirtd log. --- src/lxc/lxc_driver.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c index e905302..ef7827b 100644 --- a/src/lxc/lxc_driver.c +++ b/src/lxc/lxc_driver.c @@ -1992,6 +1992,9 @@ lxcReconnectVM(void *payload, const void *name ATTRIBUTE_UNUSED, void *opaque) virDomainObjLock(vm); + if (!virDomainObjIsActive(vm)) + goto cleanup; + priv = vm->privateData; if ((priv->monitor = lxcMonitorClient(driver, vm)) < 0) { goto cleanup; -- 1.7.4

3 7

[libvirt] [PATCH] tests: Update valgrind suppressions file
by Matthias Bolte 04 May '11

04 May '11

--- tests/.valgrind.supp | 58 +++++++++++++++++++++++++++++++++++++++++++------ 1 files changed, 51 insertions(+), 7 deletions(-) diff --git a/tests/.valgrind.supp b/tests/.valgrind.supp index 4af10b1..68cfa0c 100644 --- a/tests/.valgrind.supp +++ b/tests/.valgrind.supp @@ -258,14 +258,58 @@ Memcheck:Param capget(data) fun:capget + fun:* fun:capng_clear + fun:virClearCapabilities fun:__virExec fun:virExecWithHook - fun:virExec - fun:qemudProbeMachineTypes - fun:qemudCapsInitGuest - fun:qemudCapsInit - fun:qemudStartup - fun:virStateInitialize - fun:main +} +{ + libnlMemoryLeak1 + Memcheck:Leak + fun:malloc + fun:strdup + obj:/usr/lib/libnl.so.1.1 +} +{ + libnlMemoryLeak2 + Memcheck:Leak + fun:calloc + obj:/usr/lib/libnl.so.1.1 +} +{ + libselinuxMemoryLeak1 + Memcheck:Leak + fun:malloc + fun:getdelim + obj:/lib/libselinux.so.1 +} +{ + dashMemoryLeak1 + Memcheck:Leak + fun:malloc + obj:/bin/dash +} +{ + dashMemoryLeak2 + Memcheck:Leak + fun:malloc + fun:strdup + obj:/bin/dash +} +{ + vboxMemoryLeak1 + Memcheck:Leak + ... + fun:VBoxNsxpNS_InitXPCOM2 +} +{ + libnetcfMemoryLeak1 + fun:malloc + fun:xmlStrndup + fun:xmlHashUpdateEntry3 + fun:* + fun:xsltRegisterAllExtras + fun:drv_init + fun:interfaceOpenInterface } -- 1.7.0.4

3 5

[libvirt] [PATCH v2] Fix disability to run on systems with no PCI bus
by Michal Privoznik 03 May '11

03 May '11

The patch which moved libpciaccess initialization to one place caused regression - we were not able to run on system with no PCI bus, like s390(x). --- src/node_device/node_device_udev.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c index 2139ef3..fcff252 100644 --- a/src/node_device/node_device_udev.c +++ b/src/node_device/node_device_udev.c @@ -1421,8 +1421,12 @@ static int udevDeviceMonitorShutdown(void) ret = -1; } +#if defined __s390__ || defined __s390x_ + /* Nothing was initialized, nothing needs to be cleaned up */ +#else /* pci_system_cleanup returns void */ pci_system_cleanup(); +#endif return ret; } @@ -1595,6 +1599,10 @@ static int udevDeviceMonitorStartup(int privileged) int ret = 0; int pciret; +#if defined __s390__ || defined __s390x_ + /* On x390(x) system there is no PCI bus. + * Therefore there is nothing to initialize here. */ +#else if ((pciret = pci_system_init()) != 0) { /* Ignore failure as non-root; udev is not as helpful in that * situation, but a non-privileged user won't benefit much @@ -1607,6 +1615,7 @@ static int udevDeviceMonitorStartup(int privileged) goto out; } } +#endif if (VIR_ALLOC(priv) < 0) { virReportOOMError(); -- 1.7.4.4

3 2

[libvirt] Libvirt 0.9.1 week freeze, RC1 version to test
by Daniel Veillard 03 May '11

03 May '11

So we are entering the week freeze for the new version. I have made an rc1 tarball available at: ftp://libvirt.org/libvirt/libvirt-0.9.1-rc1.tar.gz I have tried it locally and basic operations seems to work fine, please give it a try too, thanks ! Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel(a)veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/

4 7

[libvirt] Libvirt and IPSec (was: What about Trusted Virtual Domains???)
by Paolo Smiraglia 02 May '11

02 May '11

Hi to everyone! First of all, sorry for the thread subject change. Due to the several issues of the Libvirt implementation of the Trusted Virtual Domains (TVD), I decided to approach the topic in a modular manner. I think that the first step should be to define the IPSec support or, more in general, the secure tunnel support for Libvirt. I see the implementation divided in two step: 1. define a new driver called 'sectunnel' which describes a generic secure tunnel that will be established using several technologies (for now using only ipsec) 2. modify the existing 'network' driver by adding the possibility to specify the 'sectunnel' that the network have to use in the virtual network definition As an example, you can see below a possible XML definition of the network which use a secure tunnel and the corresponding 'sectunnel' XML definition: NETWORK DEFINITION ================== <network> <name>sec-net</name> <uuid>3e3fce45-4f53-4fa7-bb32-11f34168b82b</uuid> <bridge name='virbr0' /> <domain name='example' /> ... <sectunnel name='sec-tun' /> <--(specify the 'sectunnel' to use) </network> SECTUNNEL DEFINITION ==================== <sectunnel type='ipsec'> <name>sec-tun</name> <uuid>8b7fd1b0-4463-43b7-8b6e-8006344aeb66</uuid>  <sa> <secret uuid='...' /> <--(specify the 'secret' which contains the pre-shared key) </sa>  <sp> <src_range address='10.0.0.1' prefixlen='30' port='5000' /> <dst_range address='10.0.0.2' prefixlen='30' port='5000' /> <upperspec protocol='any' /> <policy direction='out' action='ipsec'> <rule protocol='esp' mode='tunnel' level='require'> <src address='192.168.0.1' port='55055' /> <dst address='192.168.0.2' port='55055' /> </rule> </policy> </sp> <sp> <src_range address='10.0.0.2' prefixlen='30' port='5000' /> <dst_range address='10.0.0.1' prefixlen='30' port='5000' /> <upperspec protocol='any' /> <policy direction='in' action='ipsec'> <rule protocol='esp' mode='tunnel' level='require'> <src address='192.168.0.2' port='55055' /> <dst address='192.168.0.1' port='55055' /> </rule> </policy> </sp> </sectunnel> As you can see in the 'sectunnel' XML definition, I use a 'secret' element. This element is a Libvirt secret [1] and it stores the pre-shared key used by IPSec to establish the Security Associations (SA). Obviously this feature requires to define a new usage category in the 'secret' driver definition. Another possible way to establish the SA is to use the X.509 certificates. To this purpose, I think that the certificates already used by Libvirt to setup SSL/TLS remote connections, might be used. That's all! :-) What do you think about this possible IPSec implementation? Thanks in advance for the replies! Best regards, PAOLO LINK LIST --------- [1] http://libvirt.org/formatsecret.html -- PAOLO SMIRAGLIA Department of Control and Computer Engineering Mobile: +39 (333) 527 3593 Email: paolo.smiraglia(a)polito.it

3 12