- Devel - libvirt List Archives

[libvirt] [PATCH 0/8] Avoid filesystem cache pollution during virsh save
by Eric Blake 19 Jul '11

19 Jul '11

Saving a domain's state creates a large file, which risks polluting the filesystem cache and slowing down a system. If a system has a lot of domians simultaneously being saved (such as the libvirt-guests init script doing managed saves), then this can cause noticeable slowdown due to filesystem thrashing. This patch series has been successfully tested to do 'virsh save dom file --direct', with lsof(1) used to verify that O_DIRECT was in use, and the resulting file was successfully used with 'virsh restore file'. Still to come - wire up O_DIRECT on the 'virsh restore' path (yes, that means adding virDomainRestoreFlags - if only we had had the foresight to use flags everywhere). Wire up qemu.conf to allow the automatic use of --direct on automatic core dumps. Wire up libvirt-guests init script to allow the use of --direct. But I had enough in place to get the review started now. Also, this series demonstrates some of the points I was making about adding a new Flags API in this thread: https://www.redhat.com/archives/libvir-list/2011-July/msg00762.html This series requires and was tested on top of these (un-acked) patches: https://www.redhat.com/archives/libvir-list/2011-July/msg00670.html https://www.redhat.com/archives/libvir-list/2011-July/msg00675.html Eric Blake (8): save: document new public API save: wire up remote protocol save: wire up trivial saveFlags implementations save: add --direct flag to virsh save operations save: let iohelper handle inherited fd save: let iohelper work on O_DIRECT fds save: add virDirectFd wrapper type save: support O_DIRECT during qemu saves cfg.mk | 1 + configure.ac | 6 +- include/libvirt/libvirt.h.in | 14 +++ po/POTFILES.in | 1 + src/Makefile.am | 1 + src/driver.h | 6 ++ src/fdstream.c | 32 +++----- src/libvirt.c | 94 +++++++++++++++++++++- src/libvirt_private.syms | 6 ++ src/libvirt_public.syms | 5 + src/libxl/libxl_driver.c | 17 ++++- src/qemu/qemu_driver.c | 76 +++++++++++++----- src/remote/remote_driver.c | 1 + src/remote/remote_protocol.x | 11 +++- src/remote_protocol-structs | 6 ++ src/test/test_driver.c | 20 ++++- src/util/iohelper.c | 185 +++++++++++++++++++++++++++++++---------- src/util/virdirect.c | 149 +++++++++++++++++++++++++++++++++ src/util/virdirect.h | 37 +++++++++ src/vbox/vbox_tmpl.c | 19 ++++- src/xen/xen_driver.c | 17 ++++- tools/virsh.c | 12 +++- tools/virsh.pod | 17 +++- 23 files changed, 632 insertions(+), 101 deletions(-) create mode 100644 src/util/virdirect.c create mode 100644 src/util/virdirect.h -- 1.7.4.4

2 44

Re: [libvirt] mingw: test-poll pipe part fails
by Eric Blake 19 Jul '11

19 Jul '11

[adding libvirt] On 06/04/2011 12:24 AM, Paolo Bonzini wrote: > On Sat, Jun 4, 2011 at 00:37, Matthias Bolte > <matthias.bolte(a)googlemail.com> wrote: >> After testing a while and reading MSDN docs the problem seems to be >> that MsgWaitForMultipleObjects doesn't work on pipes. It doesn't >> actually wait but just returns immediately. Digging MSDN and googling >> about this suggest that there is no simple solution to this. > > Yes, Windows pipes are that broken. :( > > Using socketpair is a possibly good idea, but I would do it on > libvirtd only. I don't know exactly how libvirtd uses this pipe, but > perhaps it can be changed to an eventfd-like abstraction that can be > used with both Windows and Unix. This is similar to Eric's > suggestion, but without the pipe at all. It would also be a > libvirtd-specific suggestion. I'm wondering if the problem here is that libvirt is trying to use the pipe-to-self mechanism as a fundamental event loop idiom. That is, the reason libvirt is calling poll is in order to minimize CPU until something interesting happens, where interesting includes needing to wake up a helper thread to do an action inside locks in response to the receipt of a signal. Maybe you are on to something, and replacing all uses of pipe() with virPipeToSelf() (which uses pipe() for efficiency on Linux, but socketpair() on mingw), would allow libvirt to continue to use the pipe-to-self idiom while also using fds that can actually be poll'd on mingw. -- Eric Blake eblake(a)redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org

4 6

[libvirt] enumerating disk devices
by Serge E. Hallyn 19 Jul '11

19 Jul '11

Hi, Some people appear to have autostart VMs residing on slow storage. If libvirtd starts too early, it'll simply fail trying to start those VMs. It'd be nice to know when all the storage on which autostart containers depend becomes available, so as to safely start libvirt. The python script appended below is one approach. It could be called as something like for f in /etc/libvirt/qemu/autostart/* /etc/libvirt/lxc/autostart/*; do for d in `libvirt_list_storage.py $f`; do if [ ! -r $d ]; then echo "storage not yet ready" exit 1 fi done done echo "starting libvirtd" libvirtd (Realistically I would have this running in an upstart job which is 'start on mounted' and, if all storage available, emits a 'libvirt-storage-up' event). Are there better ways to go about this? thanks, -serge #!/usr/bin/python import sys from xml.dom.minidom import parse dom = parse(sys.argv[1]) def getText(nodelist): rc = [] for node in nodelist: if node.nodeType == node.TEXT_NODE: rc.append(node.data) return ''.join(rc) def print_domname(dom): #return print "Domain %s\n" % getText(dom.childNodes) def liststorages(dom): print_domname(dom.getElementsByTagName("name")[0]) disks = dom.getElementsByTagName("disk") list_disks(disks) filesystems = dom.getElementsByTagName('filesystem') list_filesystems(filesystems) def list_filesystems(fslist): for fs in fslist: if not fs.hasAttribute('type'): continue if fs.getAttribute('type') != 'mount': continue dirs = fs.getElementsByTagName("source") if len(dirs) == 0: continue if not dirs[0].getAttribute('dir'): continue print "dir: %s\n" % dirs[0].getAttribute('dir') def list_disks(disks): for disk in disks: if not disk.hasAttribute('type'): continue if disk.getAttribute('type') == 'file': list_disk_type_file(disk) elif disk.getAttribute('type') == 'block': list_disk_type_block(disk) else: # not supported continue def list_disk_type_file(disk): disksrcs = disk.getElementsByTagName("source") if len(disksrcs) == 0: return if not disksrcs[0].hasAttribute('file'): return print "file: %s\n" % disksrcs[0].getAttribute('file') def list_disk_type_block(disk): devsrscs = disk.getElementsByTagName("source") if len(devsrscs) == 0: return if not devsrscs[0].hasAttribute('dev'): return print "dev: %s\n" % devsrscs[0].getAttribute('dev') liststorages(dom)

1 0

[libvirt] [PATCH 0/4] Some RPC event handler ref counting fixes
by Daniel P. Berrange 19 Jul '11

19 Jul '11

Wen previously posted a report of a crash with migration. I could not reproduce it at the time, but since then someone else did and pointed out that it only occurs when TLS is in use https://www.redhat.com/archives/libvir-list/2011-July/msg00547.html This series of patches make the event handler reference counting more robust to avoid this problem

2 9

[libvirt] [PATCH 1/2] Add some basic sanity checking of certificates before use
by Daniel P. Berrange 19 Jul '11

19 Jul '11

If the libvirt daemon or libvirt client is configured with bogus certificates, it is very unhelpful to only find out about this when a TLS connection is actually attempted. Not least because the error messages you get back for failures are incredibly obscure. This adds some basic sanity checking of certificates at the time the virNetTLSContext object is created. This is at libvirt startup, or when creating a virNetClient instance. This checks that the certificate expiry/start dates are valid and that the certificate is actually signed by the CA that is loaded. * src/rpc/virnettlscontext.c: Add certificate sanity checks --- src/rpc/virnettlscontext.c | 149 ++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 145 insertions(+), 4 deletions(-) diff --git a/src/rpc/virnettlscontext.c b/src/rpc/virnettlscontext.c index 4fa6fbb..b0dcfe5 100644 --- a/src/rpc/virnettlscontext.c +++ b/src/rpc/virnettlscontext.c @@ -67,6 +67,7 @@ struct _virNetTLSSession { bool handshakeComplete; + bool isServer; char *hostname; gnutls_session_t session; virNetTLSSessionWriteFunc writeFunc; @@ -95,6 +96,134 @@ static void virNetTLSLog(int level, const char *str) { VIR_DEBUG("%d %s", level, str); } + +static gnutls_x509_crt_t virNetTLSContextSanityCheckCert(bool isServer, + const char *certFile) +{ + gnutls_datum_t data; + gnutls_x509_crt_t cert = NULL; + char *buf = NULL; + int ret = -1; + time_t now; + + if ((now = time(NULL)) == ((time_t)-1)) { + virReportSystemError(errno, "%s", + _("cannot get current time")); + goto cleanup; + } + + if (gnutls_x509_crt_init(&cert) < 0) { + virNetError(VIR_ERR_SYSTEM_ERROR, "%s", + _("Unable to initialize certificate")); + goto cleanup; + } + + if (virFileReadAll(certFile, (1<<16), &buf) < 0) + goto cleanup; + + data.data = (unsigned char *)buf; + data.size = strlen(buf); + + if (gnutls_x509_crt_import(cert, &data, GNUTLS_X509_FMT_PEM) < 0) { + virNetError(VIR_ERR_SYSTEM_ERROR, isServer ? + _("Unable to import server certificate %s") : + _("Unable to import client certificate %s"), + certFile); + goto cleanup; + } + + if (gnutls_x509_crt_get_expiration_time(cert) < now) { + virNetError(VIR_ERR_SYSTEM_ERROR, isServer ? + _("The server certificate %s has expired") : + _("The client certificate %s has expired"), + certFile); + goto cleanup; + } + + if (gnutls_x509_crt_get_activation_time(cert) > now) { + virNetError(VIR_ERR_SYSTEM_ERROR, isServer ? + _("The server certificate %s is not yet active") : + _("The client certificate %s is not yet active"), + certFile); + goto cleanup; + } + + ret = 0; + +cleanup: + if (ret != 0) { + gnutls_x509_crt_deinit(cert); + cert = NULL; + } + VIR_FREE(buf); + return cert; +} + + +static int virNetTLSContextSanityCheckCredentials(bool isServer, + const char *cacertFile, + const char *certFile) +{ + gnutls_x509_crt_t cert = NULL; + gnutls_x509_crt_t cacert = NULL; + int ret = -1; + unsigned int status; + + if (access(certFile, R_OK) == 0) { + if (!(cert = virNetTLSContextSanityCheckCert(isServer, certFile))) + goto cleanup; + } + if (access(cacertFile, R_OK) == 0) { + if (!(cacert = virNetTLSContextSanityCheckCert(isServer, cacertFile))) + goto cleanup; + } + + if (cert && cacert) { + if (gnutls_x509_crt_list_verify(&cert, 1, + &cacert, 1, + NULL, 0, + 0, &status) < 0) { + virNetError(VIR_ERR_SYSTEM_ERROR, "%s", isServer ? + _("Unable to verify server certificate against CA certificate") : + _("Unable to verify client certificate against CA certificate")); + goto cleanup; + } + + if (status != 0) { + const char *reason = _("Invalid certificate"); + + if (status & GNUTLS_CERT_INVALID) + reason = _("The certificate is not trusted."); + + if (status & GNUTLS_CERT_SIGNER_NOT_FOUND) + reason = _("The certificate hasn't got a known issuer."); + + if (status & GNUTLS_CERT_REVOKED) + reason = _("The certificate has been revoked."); + +#ifndef GNUTLS_1_0_COMPAT + if (status & GNUTLS_CERT_INSECURE_ALGORITHM) + reason = _("The certificate uses an insecure algorithm"); +#endif + + virNetError(VIR_ERR_SYSTEM_ERROR, + _("Our own certificate %s failed validation against %s: %s"), + certFile, cacertFile, reason); + goto cleanup; + } + } + + ret = 0; + +cleanup: + if (cert) + gnutls_x509_crt_deinit(cert); + if (cacert) + gnutls_x509_crt_deinit(cacert); + return ret; +} + + static int virNetTLSContextLoadCredentials(virNetTLSContextPtr ctxt, bool isServer, const char *cacert, @@ -217,6 +346,10 @@ static virNetTLSContextPtr virNetTLSContextNew(const char *cacert, goto error; } + if (requireValidCert && + virNetTLSContextSanityCheckCredentials(isServer, cacert, cert) < 0) + goto error; + if (virNetTLSContextLoadCredentials(ctxt, isServer, cacert, cacrl, cert, key) < 0) goto error; @@ -574,15 +707,21 @@ static int virNetTLSContextValidCertificate(virNetTLSContextPtr ctxt, } if (gnutls_x509_crt_get_expiration_time(cert) < now) { - virNetError(VIR_ERR_SYSTEM_ERROR, "%s", - _("The client certificate has expired")); + /* Warning is reversed from what you expect, since with + * this code it is the Server checking the client and + * vica-verca */ + virNetError(VIR_ERR_SYSTEM_ERROR, "%s", sess->isServer ? + _("The client certificate has expired") : + _("The server certificate has expired")); gnutls_x509_crt_deinit(cert); goto authdeny; } if (gnutls_x509_crt_get_activation_time(cert) > now) { - virNetError(VIR_ERR_SYSTEM_ERROR, "%s", - _("The client certificate is not yet active")); + /* client/server order reversed. see above */ + virNetError(VIR_ERR_SYSTEM_ERROR, "%s", sess->isServer ? + _("The client certificate is not yet active") : + _("The server certificate is not yet active")); gnutls_x509_crt_deinit(cert); goto authdeny; } @@ -756,6 +895,8 @@ virNetTLSSessionPtr virNetTLSSessionNew(virNetTLSContextPtr ctxt, gnutls_transport_set_pull_function(sess->session, virNetTLSSessionPull); + sess->isServer = ctxt->isServer; + return sess; error: -- 1.7.4.4

2 4

Re: [libvirt] [virt-tools-list] [PATCH virt-manager v4] Add inspection to virt-manager
by Richard W.M. Jones 19 Jul '11

19 Jul '11

On Tue, Jul 19, 2011 at 10:46:49AM -0400, Cole Robinson wrote: > Maybe we can cache the png data per detected OS value rather than per > VM? Not sure if that collides with licensing issues, but would likely > mean storing less data on disk. You can't do that without getting into trademark issues. The icon that is displayed must have come from precisely the same guest. The icons are not large anyhow. One thing I meant to ask about Dan's proposal: > > $HOME/.local/libvirt/$CONN_URI/$DOMAIN_UUID/screenshot.png > > $HOME/.local/libvirt/$CONN_URI/$DOMAIN_UUID/icon.png > > $HOME/.local/libvirt/$CONN_URI/$DOMAIN_UUID/osinfo.json Do we need the connection URI? Isn't the dom UUID unique enough? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

3 2

[libvirt] [PATCH] Fix reporting of errors for p2p migration
by Daniel P. Berrange 19 Jul '11

19 Jul '11

Starting/ending jobs when closing the connection may reset any error which was reported earlier in p2p migration. We must save the original error before doing so. This means we can also just call virConnectClose as normal, instead of virUnrefConnect * src/qemu/qemu_migration.c: Preserve errors in p2p migration --- src/qemu/qemu_migration.c | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index dfa80e3..1803b9f 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2141,6 +2141,8 @@ static int doPeer2PeerMigrate(struct qemud_driver *driver, int ret = -1; virConnectPtr dconn = NULL; bool p2p; + virErrorPtr orig_err = NULL; + VIR_DEBUG("driver=%p, sconn=%p, vm=%p, xmlin=%s, dconnuri=%s, " "uri=%s, flags=%lx, dname=%s, resource=%lu", driver, sconn, vm, NULLSTR(xmlin), NULLSTR(dconnuri), @@ -2191,10 +2193,14 @@ static int doPeer2PeerMigrate(struct qemud_driver *driver, dconnuri, flags, dname, resource); cleanup: - /* don't call virConnectClose(), because that resets any pending errors */ + orig_err = virSaveLastError(); qemuDomainObjEnterRemoteWithDriver(driver, vm); - virUnrefConnect(dconn); + virConnectClose(dconn); qemuDomainObjExitRemoteWithDriver(driver, vm); + if (orig_err) { + virSetError(orig_err); + virFreeError(orig_err); + } return ret; } -- 1.7.4.4

2 1

Re: [libvirt] [virt-tools-list] [PATCH virt-manager v4] Add inspection to virt-manager
by Richard W.M. Jones 19 Jul '11

19 Jul '11

On Tue, Jul 19, 2011 at 03:40:53PM +0100, Daniel P. Berrange wrote: > On Tue, Jul 19, 2011 at 10:17:02AM -0400, Cole Robinson wrote: > > On 07/19/2011 03:49 AM, Richard W.M. Jones wrote: > > > On Mon, Jul 18, 2011 at 05:41:00PM -0400, Cole Robinson wrote: > > >> On 07/18/2011 02:53 PM, Richard W.M. Jones wrote: > > >>> This updates 1/4 with the fixes you suggested. Also, all check-pylint > > >>> warnings and errors have been fixed. > > >>> > > >>> Rich. > > >>> > > >> > > >> Thanks! Pushed the series. > > > > > > In reply to your comment on IRC: > > > > > > crobinso> rjones: so in the default usage of virt-manager in fedora, > > > the guest inspection probably won't work since we > > > save disk images in /var/lib/libvirt/images/, which > > > regular user doesn't have access to. > > > crobinso> rjones: that's not entirely clear from feature page. > > > crobinso> rjones: I'm thinking of adding a disk path access check in > > > the inspection thread, to avoid flooding the logs > > > with errors if we can't even read the disk > > > image. that should be safe to do? > > > > > > I always run virt-manager as root (or from sudo) so this hasn't been > > > an issue. What user does virt-manager run as normally? > > > > > > > I think most common usage is just running virt-manager as the logged in > > user, using policykit to authenticate the libvirt connection so the rest > > of the app doesn't have root privs. > > > > > AFAICT if there's no access to the disks, then the call to either > > > g.add_drive_opts or g.launch will throw an exception. But the > > > inspection._vmseen hash will mean this will only happen once per > > > domain per run of virt-manager. > > > > > > On an unrelated note: I think we need to cache inspection between runs > > > of virt-manager. Does virt-manager currently store permanent state (I > > > assume it must do - ie. list of connections), and where? > > > > > > > We store config like that in gconf, though I don't think sticking > > largish data like list of applications or a png in there is a good idea. > > hostname + os info could be cached, though ideally the latter would be > > stored in libvirt XML at some point. > > Agreed, it would be desirable to cache the PNGs in $HOME/.virt-manager > though, perhaps as > > $HOME/.virt-manager/icons/$CONN_URI/$DOMAIN_UUID > > To avoid growing without bound, probably want to have virt-manager > purge files in that directory on startup, if the PNG is older than > 3 months and the associated connection or domain no longer exists. > ie don't immediately purge them, since a user might later re-add a > connection or re-create a VM. > > Another thought though is that we've also got some work going on a > little plugin for gnome-shell to capture & display screenshots of > VMs. It might be nice for virt-manager to take advantage of this > too, while also the shell plugin might like the icon image. So > perhaps we should declare a standard location for storing assets > related to a VM, which are expensive to extract/fetch. > > $HOME/.local/libvirt/$CONN_URI/$DOMAIN_UUID/screenshot.png > $HOME/.local/libvirt/$CONN_URI/$DOMAIN_UUID/icon.png > $HOME/.local/libvirt/$CONN_URI/$DOMAIN_UUID/osinfo.json > > or something like that +1 CC-ing to libvir-list. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

1 0

[libvirt] RFC (V2) New virDomainBlockPull API family to libvirt
by Adam Litke 19 Jul '11

19 Jul '11

In light of discussion on V1 of this API, here is V2 of the next generation BlockPull API. In this series we make the abort and info operations more generic so that they may apply to future block jobs (such as compression, live block copy, etc). We also add in a mechanism to set the maximum bandwidth of an operation. Changes since V1: - Make virDomainBlockPullAbort() and virDomainGetBlockPullInfo() into a generic BlockJob interface. - Added virDomainBlockJobSetSpeed() - Rename VIR_DOMAIN_EVENT_ID_BLOCK_PULL event to fit into block job API - Add bandwidth argument to virDomainBlockPull() Summary of changes since first generation patch series: - Qemu dropped incremental streaming so remove libvirt incremental BlockPull() API - Rename virDomainBlockPullAll() to virDomainBlockPull() - Changes required to qemu monitor handlers for changed command names -- To help speed the provisioning process for large domains, new QED disks are created with backing to a template image. These disks are configured with copy on read such that blocks that are read from the backing file are copied to the new disk. This reduces I/O over a potentially costly path to the backing image. In such a configuration, there is a desire to remove the dependency on the backing image as the domain runs. To accomplish this, qemu will provide an interface to perform sequential copy on read operations during normal VM operation. Once all data has been copied, the disk image's link to the backing file is removed. The virDomainBlockPull API family brings this functionality to libvirt. virDomainBlockPull() instructs the hypervisor to stream the entire device in the background. Progress of this operation can be checked with the function virDomainBlockJobInfo(). An ongoing stream can be cancelled with virDomainBlockJobAbort(). virDomainBlockJobSetSpeed() allows you to limit the bandwidth that the operation may consume. An event (VIR_DOMAIN_EVENT_ID_BLOCK_JOB) will be emitted when a disk has been fully populated or if a BlockPull() operation was terminated due to an error. This event is useful to avoid polling on virDomainBlockJobInfo() for completion and could also be used by the security driver to revoke access to the backing file when it is no longer needed. /* * BlockJob API */ /** * virDomainBlockJobType: * * VIR_DOMAIN_BLOCK_JOB_TYPE_PULL: Block Pull (virDomainBlockPull) */ typedef enum { VIR_DOMAIN_BLOCK_JOB_TYPE_UNKNOWN = -1, VIR_DOMAIN_BLOCK_JOB_TYPE_PULL = 0, } virDomainBlockJobType; /* An iterator for monitoring block job operations */ typedef unsigned long long virDomainBlockJobCursor; typedef struct _virDomainBlockJobInfo virDomainBlockJobInfo; struct _virDomainBlockJobInfo { virDomainBlockJobType type; /* * The following fields provide an indication of block job progress. @cur * indicates the current position and will be between 0 and @end. @end is * the final cursor position for this operation and represents completion. * To approximate progress, divide @cur by @end. */ virDomainBlockJobCursor cur; virDomainBlockJobCursor end; }; typedef virDomainBlockJobInfo *virDomainBlockJobInfoPtr; /** * virDomainBlockJobAbort: * @dom: pointer to domain object * @path: fully-qualified filename of disk * @flags: currently unused, for future extension * * Cancel the active block job on the given disk. * * Returns -1 in case of failure, 0 when successful. */ int virDomainBlockJobAbort(virDomainPtr dom, const char *path, unsigned int flags); /** * virDomainGetBlockJobInfo: * @dom: pointer to domain object * @path: fully-qualified filename of disk * @info: pointer to a virDomainBlockJobInfo structure * @flags: currently unused, for future extension * * Request block job information for the given disk. If an operation is active * @info will be updated with the current progress. * * Returns -1 in case of failure, 0 when successful. */ int virDomainGetBlockJobInfo(virDomainPtr dom, const char *path, virDomainBlockJobInfoPtr info, unsigned int flags); /** * virDomainBlockJobSetSpeed: * @dom: pointer to domain object * @path: fully-qualified filename of disk * @bandwidth: specify bandwidth limit in Mbps * @flags: currently unused, for future extension * * Set the maximimum allowable bandwidth that a block job may consume. If * bandwidth is 0, the limit will revert to the hypervisor default. * * Returns -1 in case of failure, 0 when successful. */ int virDomainBlockJobSetSpeed(virDomainPtr dom, const char *path, unsigned long bandwidth, unsigned int flags); /** * virConnectDomainEventBlockJobStatus: * * The final status of a block job */ typedef enum { VIR_DOMAIN_BLOCK_JOB_COMPLETED = 0, VIR_DOMAIN_BLOCK_JOB_FAILED = 1, } virConnectDomainEventBlockJobStatus; /** * virConnectDomainEventBlockPullCallback: * @conn: connection object * @dom: domain on which the event occurred * @path: fully-qualified filename of the affected disk * @type: type of block job (virDomainBlockJobType) * @status: final status of the operation (virConnectDomainEventBlockPullStatus) * @opaque: callback context * * The callback signature to use when registering for an event of type * VIR_DOMAIN_EVENT_ID_BLOCK_PULL with virConnectDomainEventRegisterAny() */ typedef void (*virConnectDomainEventBlockPullCallback)(virConnectPtr conn, virDomainPtr dom, const char *path, int type, int status, void *opaque); /** * virDomainBlockPull: * @dom: pointer to domain object * @path: Fully-qualified filename of disk * @bandwidth: (optional) specify copy bandwidth limit in Mbps * @flags: currently unused, for future extension * * Populate a disk image with data from its backing image. Once all data from * its backing image has been pulled, the disk no longer depends on a backing * image. This function pulls data for the entire device in the background. * Progress of the operation can be checked with virDomainGetBlockJobInfo() and * the operation can be aborted with virDomainBlockJobAbort(). When finished, * an asynchronous event is raised to indicate the final status. * * The maximum bandwidth (in Mbps) that will be used to do the copy can be * specified with the bandwidth parameter. If set to 0, libvirt will choose a * suitable default. Some hypervisors do not support this feature and will * return an error if bandwidth is not 0. * * Returns 0 if the operation has started, -1 on failure. */ int virDomainBlockPull(virDomainPtr dom, const char *path, unsigned long bandwidth, unsigned int flags); -- Adam Litke IBM Linux Technology Center

2 1

[libvirt] [PATCH] Quieten build & ensure API build scripts exit with non-zero status
by Daniel P. Berrange 19 Jul '11

19 Jul '11

From: "Daniel P. Berrange" <berrange(a)redhat.com> The current API build scripts will continue and exit with a zero status even if they find problems. This has been the cause of many build problems, or hidden build errors, in the past. Change the scripts so they always exit with a non-zero status for any problems they do not understand. Also turn off all debug output by default so they respect $(AM_V_GEN) * docs/Makefile.am: Use $(AM_V_GEN) for API/HTML scripts * docs/apibuild.py, python/generator.py: Exit with non-zero status if problems are found. Also be silent, not outputting any debug messages. * src/Makefile.am: Use $(AM_V_GEN) for ESX generator * python/Makefile.am: Tweak rule --- docs/Makefile.am | 11 ++---- docs/apibuild.py | 95 ++++++++++++++++++++++++++++++-------------------- python/Makefile.am | 6 ++-- python/generator.py | 16 +++++--- src/Makefile.am | 2 +- 5 files changed, 75 insertions(+), 55 deletions(-) diff --git a/docs/Makefile.am b/docs/Makefile.am index 7ff94a0..4ba6b9d 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -153,21 +153,18 @@ internals/%.html.tmp: internals/%.html.in subsite.xsl page.xsl sitemap.html.in html/index.html: libvirt-api.xml newapi.xsl page.xsl sitemap.html.in - -@if [ -x $(XSLTPROC) ] ; then \ - echo "Rebuilding the HTML pages from the XML API" ; \ + $(AM_V_GEN)if [ -x $(XSLTPROC) ] ; then \ $(XSLTPROC) --nonet -o $(srcdir)/ \ - $(srcdir)/newapi.xsl $(srcdir)/libvirt-api.xml ; fi - -@if test -x $(XMLLINT) && test -x $(XMLCATALOG) ; then \ + $(srcdir)/newapi.xsl $(srcdir)/libvirt-api.xml ; fi ; \ + if test -x $(XMLLINT) && test -x $(XMLCATALOG) ; then \ if $(XMLCATALOG) '$(XML_CATALOG_FILE)' "-//W3C//DTD XHTML 1.0 Strict//EN" \ > /dev/null ; then \ - echo "Validating the resulting XHTML pages" ; \ SGML_CATALOG_FILES='$(XML_CATALOG_FILE)' \ $(XMLLINT) --catalogs --nonet --valid --noout $(srcdir)/html/*.html ; \ else echo "missing XHTML1 DTD" ; fi ; fi $(addprefix $(srcdir)/,$(devhelphtml)): $(srcdir)/libvirt-api.xml $(devhelpxsl) - -@echo Rebuilding devhelp files - -@if [ -x $(XSLTPROC) ] ; then \ + $(AM_V_GEN)if [ -x $(XSLTPROC) ] ; then \ $(XSLTPROC) --nonet -o $(srcdir)/devhelp/ \ $(top_srcdir)/docs/devhelp/devhelp.xsl $(srcdir)/libvirt-api.xml ; fi diff --git a/docs/apibuild.py b/docs/apibuild.py index f31a853..cb273dc 100755 --- a/docs/apibuild.py +++ b/docs/apibuild.py @@ -11,6 +11,8 @@ import os, sys import string import glob +quiet=1 +warnings=0 debug=0 debugsym=None @@ -95,7 +97,7 @@ class identifier: self.conditionals = None else: self.conditionals = conditionals[:] - if self.name == debugsym: + if self.name == debugsym and not quiet: print "=> define %s : %s" % (debugsym, (module, type, info, extra, conditionals)) @@ -155,7 +157,7 @@ class identifier: def update(self, header, module, type = None, info = None, extra=None, conditionals=None): - if self.name == debugsym: + if self.name == debugsym and not quiet: print "=> update %s : %s" % (debugsym, (module, type, info, extra, conditionals)) if header != None and self.header == None: @@ -203,7 +205,7 @@ class index: if d != None and name != None and type != None: self.references[name] = d - if name == debugsym: + if name == debugsym and not quiet: print "New ref: %s" % (d) return d @@ -244,9 +246,9 @@ class index: elif type == "macro": self.macros[name] = d else: - print "Unable to register type ", type + self.warning("Unable to register type ", type) - if name == debugsym: + if name == debugsym and not quiet: print "New symbol: %s" % (d) return d @@ -260,8 +262,8 @@ class index: if self.macros.has_key(id): del self.macros[id] if self.functions.has_key(id): - print "function %s from %s redeclared in %s" % ( - id, self.functions[id].header, idx.functions[id].header) + self.warning("function %s from %s redeclared in %s" % ( + id, self.functions[id].header, idx.functions[id].header)) else: self.functions[id] = idx.functions[id] self.identifiers[id] = idx.functions[id] @@ -273,15 +275,15 @@ class index: if self.macros.has_key(id): del self.macros[id] if self.variables.has_key(id): - print "variable %s from %s redeclared in %s" % ( - id, self.variables[id].header, idx.variables[id].header) + self.warning("variable %s from %s redeclared in %s" % ( + id, self.variables[id].header, idx.variables[id].header)) else: self.variables[id] = idx.variables[id] self.identifiers[id] = idx.variables[id] for id in idx.structs.keys(): if self.structs.has_key(id): - print "struct %s from %s redeclared in %s" % ( - id, self.structs[id].header, idx.structs[id].header) + self.warning("struct %s from %s redeclared in %s" % ( + id, self.structs[id].header, idx.structs[id].header)) else: self.structs[id] = idx.structs[id] self.identifiers[id] = idx.structs[id] @@ -294,8 +296,8 @@ class index: self.identifiers[id] = idx.unions[id] for id in idx.typedefs.keys(): if self.typedefs.has_key(id): - print "typedef %s from %s redeclared in %s" % ( - id, self.typedefs[id].header, idx.typedefs[id].header) + self.warning("typedef %s from %s redeclared in %s" % ( + id, self.typedefs[id].header, idx.typedefs[id].header)) else: self.typedefs[id] = idx.typedefs[id] self.identifiers[id] = idx.typedefs[id] @@ -311,15 +313,15 @@ class index: if self.enums.has_key(id): continue if self.macros.has_key(id): - print "macro %s from %s redeclared in %s" % ( - id, self.macros[id].header, idx.macros[id].header) + self.warning("macro %s from %s redeclared in %s" % ( + id, self.macros[id].header, idx.macros[id].header)) else: self.macros[id] = idx.macros[id] self.identifiers[id] = idx.macros[id] for id in idx.enums.keys(): if self.enums.has_key(id): - print "enum %s from %s redeclared in %s" % ( - id, self.enums[id].header, idx.enums[id].header) + self.warning("enum %s from %s redeclared in %s" % ( + id, self.enums[id].header, idx.enums[id].header)) else: self.enums[id] = idx.enums[id] self.identifiers[id] = idx.enums[id] @@ -330,10 +332,10 @@ class index: # check that function condition agrees with header if idx.functions[id].conditionals != \ self.functions[id].conditionals: - print "Header condition differs from Function for %s:" \ - % id - print " H: %s" % self.functions[id].conditionals - print " C: %s" % idx.functions[id].conditionals + self.warning("Header condition differs from Function for %s:" \ + % id) + self.warning(" H: %s" % self.functions[id].conditionals) + self.warning(" C: %s" % idx.functions[id].conditionals) up = idx.functions[id] self.functions[id].update(None, up.module, up.type, up.info, up.extra) # else: @@ -356,12 +358,13 @@ class index: def analyze(self): - self.analyze_dict("functions", self.functions) - self.analyze_dict("variables", self.variables) - self.analyze_dict("structs", self.structs) - self.analyze_dict("unions", self.unions) - self.analyze_dict("typedefs", self.typedefs) - self.analyze_dict("macros", self.macros) + if not quiet: + self.analyze_dict("functions", self.functions) + self.analyze_dict("variables", self.variables) + self.analyze_dict("structs", self.structs) + self.analyze_dict("unions", self.unions) + self.analyze_dict("typedefs", self.typedefs) + self.analyze_dict("macros", self.macros) class CLexer: """A lexer for the C language, tokenize the input by reading and @@ -621,6 +624,8 @@ class CParser: info, extra, self.conditionals) def warning(self, msg): + global warnings + warnings = warnings + 1 if self.no_error: return print msg @@ -1802,7 +1807,8 @@ class CParser: return token def parse(self): - self.warning("Parsing %s" % (self.filename)) + if not quiet: + print "Parsing %s" % (self.filename) token = self.token() while token != None: if token[0] == 'name': @@ -1869,7 +1875,8 @@ class docBuilder: pass def analyze(self): - print "Project %s : %d headers, %d modules" % (self.name, len(self.headers.keys()), len(self.modules.keys())) + if not quiet: + print "Project %s : %d headers, %d modules" % (self.name, len(self.headers.keys()), len(self.modules.keys())) self.idx.analyze() def scanHeaders(self): @@ -1995,7 +2002,7 @@ class docBuilder: else: output.write(" <field name='%s' type='%s' info='%s'/>\n" % (field[1] , field[0], desc)) except: - print "Failed to serialize struct %s" % (name) + self.warning("Failed to serialize struct %s" % (name)) output.write(" </struct>\n") else: output.write("/>\n"); @@ -2023,7 +2030,7 @@ class docBuilder: def serialize_function(self, output, name): id = self.idx.functions[name] - if name == debugsym: + if name == debugsym and not quiet: print "=>", id output.write(" <%s name='%s' file='%s' module='%s'>\n" % (id.type, @@ -2059,7 +2066,7 @@ class docBuilder: output.write(" <arg name='%s' type='%s' info='%s'/>\n" % (param[1], param[0], escape(param[2]))) self.indexString(name, param[2]) except: - print "Failed to save function %s info: " % name, `id.info` + self.warning("Failed to save function %s info: " % name, `id.info`) output.write(" </%s>\n" % (id.type)) def serialize_exports(self, output, file): @@ -2074,7 +2081,7 @@ class docBuilder: escape(dict.info[data]), string.lower(data))) except: - print "Header %s lacks a %s description" % (module, data) + self.warning("Header %s lacks a %s description" % (module, data)) if dict.info.has_key('Description'): desc = dict.info['Description'] if string.find(desc, "DEPRECATED") != -1: @@ -2287,7 +2294,8 @@ class docBuilder: def serialize(self): filename = "%s/%s-api.xml" % (self.path, self.name) - print "Saving XML description %s" % (filename) + if not quiet: + print "Saving XML description %s" % (filename) output = open(filename, "w") output.write('<?xml version="1.0" encoding="ISO-8859-1"?>\n') output.write("<api name='%s'>\n" % self.name) @@ -2323,7 +2331,8 @@ class docBuilder: output.close() filename = "%s/%s-refs.xml" % (self.path, self.name) - print "Saving XML Cross References %s" % (filename) + if not quiet: + print "Saving XML Cross References %s" % (filename) output = open(filename, "w") output.write('<?xml version="1.0" encoding="ISO-8859-1"?>\n') output.write("<apirefs name='%s'>\n" % self.name) @@ -2336,7 +2345,8 @@ def rebuild(): builder = None srcdir = os.environ["srcdir"] if glob.glob(srcdir + "/../src/libvirt.c") != [] : - print "Rebuilding API description for libvirt" + if not quiet: + print "Rebuilding API description for libvirt" dirs = [srcdir + "/../src", srcdir + "/../src/util", srcdir + "/../include/libvirt"] @@ -2344,12 +2354,13 @@ def rebuild(): dirs.append("../include/libvirt") builder = docBuilder("libvirt", srcdir, dirs, []) elif glob.glob("src/libvirt.c") != [] : - print "Rebuilding API description for libvirt" + if not quiet: + print "Rebuilding API description for libvirt" builder = docBuilder("libvirt", srcdir, ["src", "src/util", "include/libvirt"], []) else: - print "rebuild() failed, unable to guess the module" + self.warning("rebuild() failed, unable to guess the module") return None builder.scan() builder.analyze() @@ -2368,5 +2379,13 @@ if __name__ == "__main__": if len(sys.argv) > 1: debug = 1 parse(sys.argv[1]) + if warnings > 0: + sys.exit(2) + else: + sys.exit(0) else: rebuild() + if warnings > 0: + sys.exit(2) + else: + sys.exit(0) diff --git a/python/Makefile.am b/python/Makefile.am index 0edb3e4..a4c9a6b 100644 --- a/python/Makefile.am +++ b/python/Makefile.am @@ -53,11 +53,11 @@ GENERATED= libvirt-export.c \ libvirt.h \ libvirt.py -generated.stamp: $(srcdir)/$(GENERATE) $(API_DESC) - $(PYTHON) $(srcdir)/$(GENERATE) $(PYTHON) +$(GENERATE).stamp: $(srcdir)/$(GENERATE) $(API_DESC) + $(AM_V_GEN)$(PYTHON) $(srcdir)/$(GENERATE) $(PYTHON) && \ touch $@ -$(GENERATED): generated.stamp +$(GENERATED): $(GENERATE).stamp $(libvirtmod_la_OBJECTS): $(GENERATED) diff --git a/python/generator.py b/python/generator.py index c27ff73..d3a3616 100755 --- a/python/generator.py +++ b/python/generator.py @@ -11,6 +11,8 @@ import sys import string import re +quiet=1 + if __name__ == "__main__": # launched as a script srcPref = os.path.dirname(sys.argv[0]) @@ -617,7 +619,8 @@ def buildStubs(): sys.exit(1) n = len(functions.keys()) - print "Found %d functions in libvirt-api.xml" % (n) + if not quiet: + print "Found %d functions in libvirt-api.xml" % (n) py_types['pythonObject'] = ('O', "pythonObject", "pythonObject", "pythonObject") try: @@ -629,8 +632,8 @@ def buildStubs(): except IOError, msg: print file, ":", msg - - print "Found %d functions in libvirt-override-api.xml" % ( + if not quiet: + print "Found %d functions in libvirt-override-api.xml" % ( len(functions.keys()) - n) nb_wrap = 0 failed = 0 @@ -662,20 +665,21 @@ def buildStubs(): export.close() wrapper.close() - print "Generated %d wrapper functions" % nb_wrap + if not quiet: + print "Generated %d wrapper functions" % nb_wrap if unknown_types: print "Missing type converters: " for type in unknown_types.keys(): print "%s:%d " % (type, len(unknown_types[type])), - print - for f in functions_failed: print "ERROR: failed %s" % f if failed > 0: return -1 + if len(unknown_types) > 0: + return -1 return 0 ####################################################################### diff --git a/src/Makefile.am b/src/Makefile.am index 54b1ca0..875a43d 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -796,7 +796,7 @@ endif BUILT_SOURCES += $(ESX_DRIVER_GENERATED) $(ESX_DRIVER_GENERATED): $(srcdir)/esx/esx_vi_generator.input $(srcdir)/esx/esx_vi_generator.py - -srcdir=$(srcdir) $(srcdir)/esx/esx_vi_generator.py + $(AM_V_GEN)srcdir=$(srcdir) $(srcdir)/esx/esx_vi_generator.py if WITH_ESX if WITH_DRIVER_MODULES -- 1.7.6

3 5