April 2019 - Devel - libvirt List Archives

[libvirt] CfP VHPC19: HPC Virtualization-Containers: Paper due May 1, 2019 (extended)
by VHPC 19 03 Apr '19

03 Apr '19

==================================================================== CALL FOR PAPERS 14th Workshop on Virtualization in High-Performance Cloud Computing (VHPC '19) held in conjunction with the International Supercomputing Conference - High Performance, June 16-20, 2019, Frankfurt, Germany. (Springer LNCS Proceedings) ==================================================================== Date: June 20, 2019 Workshop URL: http://vhpc.org Paper Submission Deadline: May 1, 2019 (extended) Springer LNCS, rolling abstract submission Abstract/Paper Submission Link: https://edas.info/newPaper.php?c=25685 Call for Papers Containers and virtualization technologies constitute key enabling factors for flexible resource management in modern data centers, and particularly in cloud environments. Cloud providers need to manage complex infrastructures in a seamless fashion to support the highly dynamic and heterogeneous workloads and hosted applications customers deploy. Similarly, HPC environments have been increasingly adopting techniques that enable flexible management of vast computing and networking resources, close to marginal provisioning cost, which is unprecedented in the history of scientific and commercial computing. Various virtualization-containerization technologies contribute to the overall picture in different ways: machine virtualization, with its capability to enable consolidation of multiple underutilized servers with heterogeneous software and operating systems (OSes), and its capability to live-migrate a fully operating virtual machine (VM) with a very short downtime, enables novel and dynamic ways to manage physical servers; OS-level virtualization (i.e., containerization), with its capability to isolate multiple user-space environments and to allow for their coexistence within the same OS kernel, promises to provide many of the advantages of machine virtualization with high levels of responsiveness and performance; lastly, unikernels provide for many virtualization benefits with a minimized OS/library surface. I/O Virtualization in turn allows physical network interfaces to take traffic from multiple VMs or containers; network virtualization, with its capability to create logical network overlays that are independent of the underlying physical topology is furthermore enabling virtualization of HPC infrastructures. Publication Accepted papers will be published in a Springer LNCS proceedings volume. Topics of Interest The VHPC program committee solicits original, high-quality submissions related to virtualization across the entire software stack with a special focus on the intersection of HPC, containers-virtualization and the cloud. Major Topics: - HPC on Containers and VMs - Containerized applications with OS-level virtualization - Lightweight applications with Unikernels - HP-as-a-Service each major topic encompassing design/architecture, management, performance management, modeling and configuration/tooling: Design / Architecture: - Containers and OS-level virtualization (LXC, Docker, rkt, Singularity, Shifter, i.a.) - Hypervisor support for heterogeneous resources (GPUs, co-processors, FPGAs, etc.) - Hypervisor extensions to mitigate side-channel attacks ([micro-]architectural timing attacks, privilege escalation) - VM & Container trust and security models - Multi-environment coupling, system software supporting in-situ analysis with HPC simulation - Cloud reliability, fault-tolerance and high-availability - Energy-efficient and power-aware virtualization - Containers inside VMs with hypervisor isolation - Virtualization support for emerging memory technologies - Lightweight/specialized operating systems in conjunction with virtual machines - Hypervisor support for heterogeneous resources (GPUs, co-processors, FPGAs, etc.) - Novel unikernels and use cases for virtualized HPC environments - ARM-based hypervisors, ARM virtualization extensions Management: - Container and VM management for HPC and cloud environments - HPC services integration, services to support HPC - Service and on-demand scheduling & resource management - Dedicated workload management with VMs or containers - Workflow coupling with VMs and containers - Unikernel, lightweight VM application management - Environments and tools for operating containerized environments (batch, orchestration) - Novel models for non-HPC workload provisioning on HPC resources Performance Measurements and Modeling: - Performance improvements for or driven by unikernels - Optimizations of virtual machine monitor platforms and hypervisors - Scalability analysis of VMs and/or containers at large scale - Performance measurement, modeling and monitoring of virtualized/cloud workloads - Virtualization in supercomputing environments, HPC clusters, HPC in the cloud Configuration / Tooling: - Tool support for unikernels: configuration/build environments, debuggers, profilers - Job scheduling/control/policy and container placement in virtualized environments - Operating MPI in containers/VMs and Unikernels - Software defined networks and network virtualization - GPU virtualization operationalization The Workshop on Virtualization in High-Performance Cloud Computing (VHPC) aims to bring together researchers and industrial practitioners facing the challenges posed by virtualization in order to foster discussion, collaboration, mutual exchange of knowledge and experience, enabling research to ultimately provide novel solutions for virtualized computing systems of tomorrow. The workshop will be one day in length, composed of 20 min paper presentations, each followed by 10 min discussion sections, plus lightning talks that are limited to 5 minutes. Presentations may be accompanied by interactive demonstrations. Important Dates May 1, 2019 - Abstract/Paper extended submission deadline (Springer LNCS) May 20, 2019 - Acceptance notification June 20th, 2019 - Workshop Day July 10th, 2019 - Camera-ready version due Chair Michael Alexander (chair), University of Vienna, Austria Anastassios Nanos (co-chair), SunLight.io, UK Andrew Younge (co-chair), Sandia National Laboratories Program committee Stergios Anastasiadis, University of Ioannina, Greece Jakob Blomer, CERN, Europe Eduardo César, Universidad Autonoma de Barcelona, Spain Taylor Childers, Argonne National Laboratory, USA Stephen Crago, USC ISI, USA Tommaso Cucinotta, St. Anna School of Advanced Studies, Italy Christoffer Dall, Columbia University, USA François Diakhaté, CEA, France Patrick Dreher, MIT, USA Kyle Hale, Northwestern University, USA Bob Killen, University of Michigan, USA Brian Kocoloski, Washington University, USA John Lange, University of Pittsburgh, USA Giuseppe Lettieri, University of Pisa, Italy Qing Liu, Oak Ridge National Laboratory, USA Nikos Parlavantzas, IRISA, France Kevin Pedretti, Sandia National Laboratories, USA Amer Qouneh, Western New England University, USA Carlos Reaño, Queen’s University Belfast, UK Borja Sotomayor, University of Chicago, USA Jonathan Sparks, Cray, USA Joe Stubbs, Texas Advanced Computing Center, USA Anata Tiwari, San Diego Supercomputer Center, USA Kurt Tutschku, Blekinge Institute of Technology, Sweden John Walters, USC ISI, USA Yasuhiro Watashiba, Osaka University, Japan Chao-Tung Yang, Tunghai University, Taiwan Na Zhang, VMware, USA Paper Submission-Publication Papers submitted to the workshop will be reviewed by at least two members of the program committee and external reviewers. Submissions should include abstract, keywords, the e-mail address of the corresponding author, and must not exceed 10 pages, including tables and figures at a main font size no smaller than 11 point. Submission of a paper should be regarded as a commitment that, should the paper be accepted, at least one of the authors will register and attend the conference to present the work. Accepted papers will be published in a Springer LNCS volume. The format must be according to the Springer LNCS Style. Initial submissions are in PDF; authors of accepted papers will be requested to provide source files. Format Guidelines: ftp://ftp.springernature.com/cs-proceeding/llncs/llncs2e.zip Abstract, Paper Submission Link: https://edas.info/newPaper.php?c=25685 Lightning Talks Lightning Talks are non-paper track, synoptical in nature and are strictly limited to 5 minutes. They can be used to gain early feedback on ongoing research, for demonstrations, to present research results, early research ideas, perspectives and positions of interest to the community. Submit abstract via the main submission link. General Information The workshop is one day in length and will be held in conjunction with the International Supercomputing Conference - High Performance (ISC) 2019, June 16-20, Frankfurt, Germany.

1 0

[libvirt] Release of libvirt-5.2.0
by Daniel Veillard 03 Apr '19

03 Apr '19

It's out, a bit late but the signed tarball and rpms are at the usual place and it's tagged in git: ftp://libvirt.org/libvirt/ I also pushed the python bindings, with only a single patch since 5.1.0 at: ftp://libvirt.org/libvirt/python/ Note that it's a release with rather large changes, new features and removal of some deprecated init scripts, and as usual a large amount of improvement and bug fixes: New features: - Add Storage Pool Capabilities output Add support to list an enumerated list of supported Storage Pools via the virConnectGetCapabilities API when connected via a Storage Driver. Add support to get a more detailed list XML output Storage Pool Capabilities vis the virConnectGetStoragePoolCapabilites API. - qemu: Support virtio-{non-}transitional device models virtio-transitional and virtio-non-transitional model values were added to the QEMU driver for the following devices: disk, interface, filesystem, rng, vsock, memballoon, controller type scsi, controller type virtio-serial, input bus virtio type passthrough, hostdev type scsi_host. These new models can be used to give fine grained control over what virtio device version is presented to the guest. - qemu: Enable firmware autoselection Libvirt allows users to provide loader path for some time now. However, this puts some burden on users because they need to know what firmware meets their requirements. Now that QEMU ships firmware description files this burden can be moved onto libvirt. It is as easy as setting the firmware attribute in the os element (accepted values are bios and efi). Moreover, libvirt automatically enables domain features needed for firmware it chooses. - snapshots: Add support for topological listings A new flag VIR_DOMAIN_SNAPSHOT_LIST_TOPOLOGICAL is available for the various snapshot listing APIs such as virDomainListAllSnapshots(). For drivers that support the flag, the listed snapshots are guaranteed to be sorted such that parents occur before children. - Xen: Add support for max grant frames setting Add support for Xen's max_grant_frames setting by adding a new xenbus controller type with a maxGrantFrames attribute. E.g. <controller type='xenbus' maxGrantFrames='64'/> - qemu: Add support for parallel migration With QEMU 4.0.0 libvirt can enable parallel migration which causes the memory pages to be processed in parallel by several threads and sent to the destination host using several connections at the same time. This may increase migration speed in case a single thread is unable to saturate the network link. Removed features: - Drop support for Upstart and "Red Hat" init scripts Not a single one of the platforms we target still uses Upstart, and the Upstart project itself has been abandoned for several years now; the same is true for the "Red Hat" (really System V) init scripts, since RHEL 7 and later releases use systemd. Improvements: - Report class information for PCI node device capability. - Split setup of IPv4 and IPv6 top level chain The requirement resulting from private chains improvement done in v5.1.0 was refined so that only tables from corresponding IP version are required. This means that if a network doesn't have IPv6 enabled then those tables are not required. - Don't default to building the QEMU driver Historically, the QEMU driver has been special in that it was enabled by default, with the option to explicitly opt-out of it; starting now, we're enabling it opportunistically if we detect that all requirements are available, just like we do with other drivers. Bug fixes: - virt-host-validate: Fix IOMMU check on s390x - qemu: Allow creating pSeries guests with graphics and no USB mouse It's now possible to prevent libvirt from automatically adding a USB mouse to pSeries guests by including a USB tablet in the input XML: doing so is desiderable as using a tablet results in a much better user experience when working with GUIs. - qemu: Set $HOME and XGD variables for qemu:///system guests This avoids files being accidentally created under / or the guests not being able to start because they lack the necessary permissions to write to that location. Thanks everybody for your contributions for this release, be it with bug reports, patches, reviews, documentation, etc... Enjoy the release ! Daniel -- Daniel Veillard | Red Hat Developers Tools http://developer.redhat.com/ veillard(a)redhat.com | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | virtualization library http://libvirt.org/

3 3

[libvirt] [perl PATCH] rpm: remove BuildRoot setting
by Daniel P. Berrangé 03 Apr '19

03 Apr '19

This is obsolete and removed from all Fedora RPMs Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- perl-Sys-Virt.spec.PL | 1 - 1 file changed, 1 deletion(-) Pushed as a trivial sync back from Fedora dist-git diff --git a/perl-Sys-Virt.spec.PL b/perl-Sys-Virt.spec.PL index b489a41..5623fe5 100644 --- a/perl-Sys-Virt.spec.PL +++ b/perl-Sys-Virt.spec.PL @@ -33,7 +33,6 @@ Summary: Represent and manage a libvirt hypervisor connection License: GPLv2+ or Artistic URL: https://metacpan.org/release/Sys-Virt Source0: https://cpan.metacpan.org/authors/id/D/DA/DANBERR/Sys-Virt-v%{version}.tar.… -BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) # Build BuildRequires: coreutils BuildRequires: findutils -- 2.20.1

1 0

[libvirt] [PATCH 0/5] util: Remove some obsolete and dead code
by Peter Krempa 03 Apr '19

03 Apr '19

Peter Krempa (5): util: Remove virPipeReadUntilEOF vmx: Remove unused variable in virVMXParseConfig vmx: Refactor number parsing in virVMXParseConfig rpc: ssh: Use virStrToLong_i instead of virParseNumber util: Remove virParseNumber src/libvirt_private.syms | 2 - src/rpc/virnetlibsshsession.c | 6 +- src/util/virutil.c | 113 ---------------------------------- src/util/virutil.h | 4 -- src/vmx/vmx.c | 62 ++++++------------- 5 files changed, 23 insertions(+), 164 deletions(-) -- 2.20.1

2 10

[libvirt] [PATCH] mingw: Bump minimum supported Fedora version to 28
by Andrea Bolognani 03 Apr '19

03 Apr '19

Should have been part of 2569ba133822, but clearly wasn't. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- mingw-libvirt.spec.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in index bea822fd73..8756a56a23 100644 --- a/mingw-libvirt.spec.in +++ b/mingw-libvirt.spec.in @@ -3,7 +3,7 @@ # This spec file assumes you are building on a Fedora version # that's still supported by the vendor. It may work on other distros # or versions, but no effort will be made to ensure that going forward. -%define min_fedora 26 +%define min_fedora 28 %if 0%{?fedora} && 0%{?fedora} >= %{min_fedora} %define supported_platform 1 -- 2.20.1

2 2

[libvirt] deadlock in remoteDispatchDomainUndefine vs daemonStreamHandleAbort
by Christian Ehrhardt 03 Apr '19

03 Apr '19

Hi, I happened to analyze a bug [1] report I got from a friend and for quite a while it was rather elusive. But I now finally got it reproducible [2] enough to share it with the community. The TL;DR of what I see is: - an automation with python-libvirt gets a SIGINT - cleanup runs destroy and further undefine - the guest closes FDs due to SIGINT and/or destroy which triggers daemonStreamHandleAbort - those two fight over the lock There I get libvirtd into a deadlock which ends up with all threads dead [4] and two of them fighting [3] (details) in particular. The to related stacks summarized are like: daemonStreamHandleWrite (failing to write) -> daemonStreamHandleAbort (closing things and cleaning up) -> ... virChrdevFDStreamCloseCb virMutexLock(&priv->devs->lock); # there is code meant to avoid such issues emitting "Unable to close" if a lock is held # but the log doesn't show this triggering with debug enabled #10 seems triggered via an "undefine" call remoteDispatchDomainUndefine ... -> virChrdevFree ... -> virFDStreamSetInternalCloseCb -> virObjectLock(virFDStreamDataPtr fdst) -> virMutexLock(&obj->lock); # closing all streams of a guest (requiring the same locks) While that already feels quite close I struggle to see where exactly we'd want to fix it. But finally having a repro-script [2] I hope that someone else here might be able to help me with that. After all it is a race - on my s390x system it triggers usually <5 tries, while on x86 I have needed up to 18 runs of the test to hang. Given different system configs it might be better or worse for you. FYI we hit this with libvirt 4.0 initially but libvirt 5.0 was just the same. I haven't built 5.1 or a recent master, but the commits since 5.0 didn't mention any issue that seems related. OTOH I'm willing and able to build and try suggestions if anyone comes up with ideas. [1]: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1822096 [2]: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1822096/+attachment/… [3]: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1822096/comments/3 [4]: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1822096/comments/17 -- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd

2 4

[libvirt] [PATCH] mingw: unconditionally enable libssh2 in RPM
by Daniel P. Berrangé 03 Apr '19

03 Apr '19

The libssh2 support in libvirt is not solely for phyp, it is used by the remote driver too. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- mingw-libvirt.spec.in | 2 -- 1 file changed, 2 deletions(-) diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in index 9983a5013f..913aaa8751 100644 --- a/mingw-libvirt.spec.in +++ b/mingw-libvirt.spec.in @@ -85,10 +85,8 @@ BuildRequires: gettext-devel BuildRequires: libtool %endif -%if %{with_phyp} BuildRequires: mingw32-libssh2 BuildRequires: mingw64-libssh2 -%endif %if %{with_esx} BuildRequires: mingw32-curl BuildRequires: mingw64-curl -- 2.20.1

2 1

[libvirt] [PATCH] mingw: remove obsolete Group tag in RPM spec
by Daniel P. Berrangé 03 Apr '19

03 Apr '19

Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- mingw-libvirt.spec.in | 1 - 1 file changed, 1 deletion(-) diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in index bea822fd73..9983a5013f 100644 --- a/mingw-libvirt.spec.in +++ b/mingw-libvirt.spec.in @@ -39,7 +39,6 @@ Release: 1%{?dist} Summary: MinGW Windows libvirt virtualization library License: LGPLv2+ -Group: Development/Libraries URL: https://libvirt.org/ %if %(echo %{version} | grep -q "\.0$"; echo $?) == 1 -- 2.20.1

2 1

[libvirt] [PATCH v3 0/6] Facilitate running libvirt builds via docker containers
by Daniel P. Berrangé 03 Apr '19

03 Apr '19

For a while QEMU has provided simple make rules for building QEMU inside standard docker container environments. This provides an equivalent mechanism for libvirt inspired by QEMU's. QEMU actually builds the container images on developer's machines locally. Libvirt already hosts pre-built images on quay.io, so that is used directly as it is quicker to download them than to build them locally. This also ensures the container contents match what the live CI system is using, as opposed to building an image with newer packages. NB, there is a currently a test failure with this change to travis that we don't currently see. Current travis runs a privileged docker container, but with this change we now run unprivileged containers. This shows that the "virsh-snapshot" test is trying to create a file under $HOME, and docker blocks this. Obviously needs fixing before pushing. Changed in v3: - Lots of new variables & variable renames - Rename to top level Makefile.ci - Rename make targets to have @ separator for the image - Refactor macOS travis config Changed in v2: - Drop cross-compilation patches temporarily. The cross-compiler images are not yet built on quay.io and rather than creating and bulding many more images manually, I'm working on script to automate setup of quay.io images - Many changes to the make rules to make it possible to use them from Travis CI to match its currently testing setup - Modify Travis CI config to use the new make rules for consistency Daniel P. Berrangé (6): tests: don't abort in fopen(/proc/mounts) tests: add targets for building libvirt inside docker containers travis: convert ubuntu, centos & mingw builds to use new make rules travis: use declarative syntax for homebrew packages travis: remove display of test-suite.log from macOS travis: put macOS script inline in the macOS matrix entry .gitignore | 1 + .travis.yml | 78 +++++----------- Makefile.am | 2 + Makefile.ci | 206 ++++++++++++++++++++++++++++++++++++++++++ tests/vircgroupmock.c | 6 +- 5 files changed, 235 insertions(+), 58 deletions(-) create mode 100644 Makefile.ci -- 2.20.1

3 20

[libvirt] [PATCH] virjson: drop compatibility macros
by Ján Tomko 03 Apr '19

03 Apr '19

Since commit 66460e3 dropped support for YAJL 1, we no longer need these. Signed-off-by: Ján Tomko <jtomko(a)redhat.com> --- src/util/virjson.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/src/util/virjson.c b/src/util/virjson.c index bb4052ba81..1dceb746b9 100644 --- a/src/util/virjson.c +++ b/src/util/virjson.c @@ -34,8 +34,6 @@ # include <yajl/yajl_gen.h> # include <yajl/yajl_parse.h> -# define yajl_size_t size_t -# define VIR_YAJL_STATUS_OK(status) ((status) == yajl_status_ok) #endif /* XXX fixme */ @@ -1615,7 +1613,7 @@ virJSONParserHandleBoolean(void *ctx, static int virJSONParserHandleNumber(void *ctx, const char *s, - yajl_size_t l) + size_t l) { virJSONParserPtr parser = ctx; char *str; @@ -1643,7 +1641,7 @@ virJSONParserHandleNumber(void *ctx, static int virJSONParserHandleString(void *ctx, const unsigned char *stringVal, - yajl_size_t stringLen) + size_t stringLen) { virJSONParserPtr parser = ctx; virJSONValuePtr value = virJSONValueNewStringLen((const char *)stringVal, @@ -1666,7 +1664,7 @@ virJSONParserHandleString(void *ctx, static int virJSONParserHandleMapKey(void *ctx, const unsigned char *stringVal, - yajl_size_t stringLen) + size_t stringLen) { virJSONParserPtr parser = ctx; virJSONParserStatePtr state; @@ -1824,7 +1822,7 @@ virJSONValueFromString(const char *jsonstring) /* Yajl 2 is nice enough to default to rejecting trailing garbage. */ rc = yajl_parse(hand, (const unsigned char *)jsonstring, len); - if (!VIR_YAJL_STATUS_OK(rc) || + if (rc != yajl_status_ok || yajl_complete_parse(hand) != yajl_status_ok) { unsigned char *errstr = yajl_get_error(hand, 1, (const unsigned char*)jsonstring, @@ -1935,7 +1933,7 @@ virJSONValueToBuffer(virJSONValuePtr object, { yajl_gen g; const unsigned char *str; - yajl_size_t len; + size_t len; int ret = -1; VIR_DEBUG("object=%p", object); -- 2.19.2

3 2