July 2019 - Devel - libvirt List Archives

[libvirt] [PATCH v1 00/31] Introduce NVMe support
by Michal Privoznik 02 Aug '19

02 Aug '19

These patches introduce a support for NVMe disks into libvirt. Note that even without them it is possible to use NVMe disks for your domains in two ways: 1) <hostdev/> - This is regular PCI assignment with all the drawbacks (no migration, no snapshots, ...) 2) <disk/> - Since NVMe disks are accessible via /dev/nvme* they can be assigned to domains. Problem is, because qemu is accessing /dev/nvme* the host kernel's storage stack is involved which adds significant latency [1]. Solution to this problem is to combine 1) and 2) together: - Bypass host kernel's storage stack by detaching the NVMe disk from the host (and attaching it to VFIO driver), and - Plug the NVMe disk into qemu's block layer so that all fancy features can be supported. On qemu command line this is done via: -drive file.driver=nvme,file.device=0000:01:00.0,file.namespace=1,format=raw,\ if=none,id=drive-virtio-disk0 \ -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,\ id=virtio-disk0,bootindex=1 \ You can find my patches also on my github [2]. 1: https://www.linux-kvm.org/images/4/4c/Userspace_NVMe_driver_in_QEMU_-_Fam_Z… 2: https://github.com/zippy2/libvirt/commits/nvme Michal Prívozník (31): virHostdevPreparePCIDevices: Separate out function body virHostdevReAttachPCIDevices: Separate out function body virpcimock: Move actions checking one level up Revert "virpcitest: Test virPCIDeviceDetach failure" virpcimock: Create driver_override file in device dirs virPCIDeviceAddressEqual: Fix const correctness virPCIDeviceAddressAsString: Fix const correctness virpci: Introduce virPCIDeviceAddressCopy qemuDomainDeviceDefValidateDisk: Reorder some checks schemas: Introduce disk type NVMe conf: Format and parse NVMe type disk util: Introduce virNVMeDevice module virhostdev: Include virNVMeDevice module virhostdevtest: Don't proceed to test cases if init failed virhostdevtest: s/CHECK_LIST_COUNT/CHECK_PCI_LIST_COUNT/ virpcimock: Introduce NVMe driver and devices virhostdevtest: Test virNVMeDevice assignment qemu: prepare NVMe devices too qemu: Take NVMe disks into account when calculating memlock limit virstoragefile: Introduce virStorageSourceChainHasNVMe domain_conf: Introduce virDomainDefHasNVMeDisk qemu_domain: Separate VFIO code qemu_domain: Introduce NVMe path getting helpers qemu: Create NVMe disk in domain namespace qemu: Allow NVMe disk in CGroups security_selinux: Simplify virSecuritySELinuxSetImageLabelInternal virSecuritySELinuxRestoreImageLabelInt: Don't skip non-local storage qemu_capabilities: Introduce QEMU_CAPS_DRIVE_NVME qemu: Generate command line of NVMe disks qemu: Don't leak storage perms on failure in qemuDomainAttachDiskGeneric qemu_hotplug: Prepare NVMe disks on hotplug docs/formatdomain.html.in | 45 +- docs/schemas/domaincommon.rng | 32 ++ src/conf/domain_conf.c | 160 +++++++ src/conf/domain_conf.h | 6 + src/libvirt_private.syms | 26 ++ src/qemu/qemu_block.c | 24 + src/qemu/qemu_capabilities.c | 4 + src/qemu/qemu_capabilities.h | 3 + src/qemu/qemu_cgroup.c | 59 ++- src/qemu/qemu_command.c | 4 + src/qemu/qemu_domain.c | 115 ++++- src/qemu/qemu_domain.h | 6 + src/qemu/qemu_driver.c | 4 + src/qemu/qemu_hostdev.c | 49 ++- src/qemu/qemu_hostdev.h | 10 + src/qemu/qemu_hotplug.c | 76 +++- src/qemu/qemu_migration.c | 1 + src/qemu/qemu_process.c | 7 + src/security/security_dac.c | 38 ++ src/security/security_selinux.c | 95 ++-- src/util/Makefile.inc.am | 2 + src/util/virhostdev.c | 350 +++++++++++++-- src/util/virhostdev.h | 25 ++ src/util/virnvme.c | 412 ++++++++++++++++++ src/util/virnvme.h | 89 ++++ src/util/virpci.c | 12 +- src/util/virpci.h | 8 +- src/util/virstoragefile.c | 73 ++++ src/util/virstoragefile.h | 17 + src/xenconfig/xen_xl.c | 1 + .../caps_2.12.0.aarch64.xml | 1 + .../caps_2.12.0.ppc64.xml | 1 + .../caps_2.12.0.s390x.xml | 1 + .../caps_2.12.0.x86_64.xml | 1 + .../qemucapabilitiesdata/caps_3.0.0.ppc64.xml | 1 + .../caps_3.0.0.riscv32.xml | 1 + .../caps_3.0.0.riscv64.xml | 1 + .../qemucapabilitiesdata/caps_3.0.0.s390x.xml | 1 + .../caps_3.0.0.x86_64.xml | 1 + .../qemucapabilitiesdata/caps_3.1.0.ppc64.xml | 1 + .../caps_3.1.0.x86_64.xml | 1 + .../caps_4.0.0.aarch64.xml | 1 + .../qemucapabilitiesdata/caps_4.0.0.ppc64.xml | 1 + .../caps_4.0.0.riscv32.xml | 1 + .../caps_4.0.0.riscv64.xml | 1 + .../qemucapabilitiesdata/caps_4.0.0.s390x.xml | 1 + .../caps_4.0.0.x86_64.xml | 1 + .../caps_4.1.0.x86_64.xml | 1 + .../disk-nvme.x86_64-latest.args | 52 +++ tests/qemuxml2argvdata/disk-nvme.xml | 63 +++ tests/qemuxml2argvtest.c | 1 + tests/qemuxml2xmloutdata/disk-nvme.xml | 1 + tests/qemuxml2xmltest.c | 1 + tests/virhostdevtest.c | 185 ++++++-- tests/virpcimock.c | 76 +++- tests/virpcitest.c | 32 -- tests/virpcitestdata/0000-01-00.0.config | Bin 0 -> 4096 bytes tests/virpcitestdata/0000-02-00.0.config | Bin 0 -> 4096 bytes 58 files changed, 1978 insertions(+), 204 deletions(-) create mode 100644 src/util/virnvme.c create mode 100644 src/util/virnvme.h create mode 100644 tests/qemuxml2argvdata/disk-nvme.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/disk-nvme.xml create mode 120000 tests/qemuxml2xmloutdata/disk-nvme.xml create mode 100644 tests/virpcitestdata/0000-01-00.0.config create mode 100644 tests/virpcitestdata/0000-02-00.0.config -- 2.21.0

2 75

[libvirt] [PATCH 1/1] tests/virsh-checkpoint/snapshot: changing 'sed' out filtering
by Daniel Henrique Barboza 02 Aug '19

02 Aug '19

There is a chance that the current sed filtering used in these new tests might fail in some machines due to the repetition of the 'virsh #' prompt at the same line, together with valid output that shouldn't be filtered. This is output of virsh-snapshot test in my T480 dev box: ./virsh-snapshot --- exp 2019-07-31 18:42:31.107399428 -0300 +++ out.cooked 2019-07-31 18:42:31.108399437 -0300 @@ -1,8 +1,3 @@ - - -Domain snapshot s3 created from 's3.xml' -Domain snapshot s2 created from 's2.xml' -Name: s2 Domain: test Current: yes State: running There are 3 valid lines missing. This is the unfiltered output: === out === Welcome to lt-virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # virsh # virsh # virsh # virsh # Domain snapshot s3 created from 's3.xml' virsh # Domain snapshot s2 created from 's2.xml' virsh # Name: s2 Domain: test Current: yes State: running Location: internal Parent: s3 Children: 0 Descendants: 0 Metadata: yes virsh # ============ We can see that the 3 lines being erased are being followed by the 'virsh #' prompt and the filtering is erasing those. A similar situation happens with virsh-checkpoint as well. This patch makes the 'sed' filtering less elegant and more crude than the current version, but more reliable to these outputs that may vary from each dev machine. We're also removing the blank lines in the expected output to make it less prone to errors as well. Signed-off-by: Daniel Henrique Barboza <danielhb413(a)gmail.com> --- Eric, feel free to accept this patch, tweak it, discard it and try something else or whatever. I was going to send the commit msg as a reply to your query in the ML, then realized that I might as well propose a fix for it. tests/virsh-checkpoint | 6 +----- tests/virsh-snapshot | 6 ++---- 2 files changed, 3 insertions(+), 9 deletions(-) diff --git a/tests/virsh-checkpoint b/tests/virsh-checkpoint index 75bdc293be..a3cad74f74 100755 --- a/tests/virsh-checkpoint +++ b/tests/virsh-checkpoint @@ -152,20 +152,16 @@ $abs_top_builddir/tools/virsh -c test:///default >out 2>err <<EOF || fail=1 EOF cat <<\EOF > exp || fail=1 - - Domain checkpoint c3 created from 'c3.xml' Domain checkpoint c2 created from 'c2.xml' c2 - Name: c2 Domain: test Parent: c3 Children: 0 Descendants: 0 - EOF -sed '1,/^virsh #/d; /virsh #/d' < out > out.cooked || fail=1 +sed '1,/^virsh #/d; s/virsh #\s//g; /^$/d' < out > out.cooked || fail=1 compare exp out.cooked || fail=1 cat <<EOF > exp || fail=1 diff --git a/tests/virsh-snapshot b/tests/virsh-snapshot index 20ff966a51..874093ea3c 100755 --- a/tests/virsh-snapshot +++ b/tests/virsh-snapshot @@ -201,9 +201,8 @@ $abs_top_builddir/tools/virsh -c test:///default >out 2>err <<EOF || fail=1 snapshot-info test --current EOF -cat <<\EOF > exp || fail=1 - +cat <<\EOF > exp || fail=1 Domain snapshot s3 created from 's3.xml' Domain snapshot s2 created from 's2.xml' Name: s2 @@ -215,9 +214,8 @@ Parent: s3 Children: 0 Descendants: 0 Metadata: yes - EOF -sed '1,/^virsh #/d; /virsh #/d' < out > out.cooked || fail=1 +sed '1,/^virsh #/d; s/virsh #\s//g; /^$/d' < out > out.cooked || fail=1 compare exp out.cooked || fail=1 cat <<EOF > exp || fail=1 -- 2.21.0

2 7

[libvirt] [PATCH] RFC: security: Make sure to decrease ref count label value
by Stefan Berger 02 Aug '19

02 Aug '19

I noticed that if a domain fails to restore, the ref count in the xattr 'trusted.libvirt.security.ref_selinux' keeps on increasing indefinitely and the VM will never restore even if the root cause for the restore failure has been removed. The reason seems to be that the code to decrease the ref count never gets called because the block above it fails due to virSecuritySELinuxTransactionAppend() failing. The simple solution seems to be to revert the order in which things are done. Signed-off-by: Stefan Berger <stefanb(a)linux.ibm.com> --- src/security/security_selinux.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/src/security/security_selinux.c b/src/security/security_selinux.c index ea20373a90..9fd29e9bca 100644 --- a/src/security/security_selinux.c +++ b/src/security/security_selinux.c @@ -1499,14 +1499,9 @@ virSecuritySELinuxRestoreFileLabel(virSecurityManagerPtr mgr, goto cleanup; } - if ((rc = virSecuritySELinuxTransactionAppend(path, NULL, - false, recall, true)) < 0) { - goto cleanup; - } else if (rc > 0) { - ret = 0; - goto cleanup; - } - + /* Recall the label so the ref count label decreases its counter + * even if transaction append below fails. + */ if (recall) { rc = virSecuritySELinuxRecallLabel(newpath, &fcon); if (rc == -2) { @@ -1519,6 +1514,14 @@ virSecuritySELinuxRestoreFileLabel(virSecurityManagerPtr mgr, } } + if ((rc = virSecuritySELinuxTransactionAppend(path, NULL, + false, recall, true)) < 0) { + goto cleanup; + } else if (rc > 0) { + ret = 0; + goto cleanup; + } + if (!recall || rc == -2) { if (stat(newpath, &buf) != 0) { VIR_WARN("cannot stat %s: %s", newpath, -- 2.20.1

3 3

[libvirt] [PATCH] news: add entry for new max_threads_per_process option in qemu.conf
by Jim Fehlig 01 Aug '19

01 Aug '19

Signed-off-by: Jim Fehlig <jfehlig(a)suse.com> --- docs/news.xml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/news.xml b/docs/news.xml index 82a4ec9c84..3a20f95a0d 100644 --- a/docs/news.xml +++ b/docs/news.xml @@ -74,6 +74,17 @@ which portions of a disk have changed since a point in time. </description> </change> + <change> + <summary> + qemu: Add support for overriding max threads per process limit + </summary> + <description> + systemd-based systems impose a limit on the number of threads a + process can spawn, which in some cases can be exceeded by qemu + processes running VMs. Add a <code>max_threads_per_process</code> + option to qemu.conf to override the system default. + </description> + </change> </section> <section title="Removed features"> <change> -- 2.22.0

2 1

[libvirt] [jenkins-ci PATCH 0/6] Update libosinfo (& friends) dependencies
by Fabiano Fidêncio 31 Jul '19

31 Jul '19

libsoup & mingw-libsoup are new dependencies for both libosinfo & osinfo-db-tools since v1.6.0 release libcurl & mingw-curl are no longer a dependency for libosinfo since v1.5.0 release Fabiano Fidêncio (6): mappings: Add mingw*-libsoup mingw*-libosinfo: Add mingw*-libsoup as dependency mingw*-osinfo-db-tools: Add mingw*-libsoup as dependency osinfo-db-tools: Add libsoup as dependency mingw*-libosinfo: Remove mingw*-curl dependency ibosinfo: Remove libcurl dependency guests/vars/mappings.yml | 6 ++++++ guests/vars/projects/libosinfo+mingw32.yml | 2 +- guests/vars/projects/libosinfo+mingw64.yml | 2 +- guests/vars/projects/libosinfo.yml | 1 - guests/vars/projects/osinfo-db-tools+mingw32.yml | 1 + guests/vars/projects/osinfo-db-tools+mingw64.yml | 1 + guests/vars/projects/osinfo-db-tools.yml | 1 + 7 files changed, 11 insertions(+), 3 deletions(-) -- 2.21.0

3 14

[libvirt] [PATCH v2 0/3] misc virhostdevs cleanups
by Daniel Henrique Barboza 31 Jul '19

31 Jul '19

changes in v2: - changed the parameter order in the function calls - gave up on moving virPCIDeviceSetX(pci, true) calls to virPCIDeviceReattach(). The attributes being set changes the behavior of virPCIDeviceReattach() in a more complex way than I expected. I still believe a simplification can be done there, but it became out of scope for a more simplistic cleanup such as this one. These are cleanups that I made together with an attempt to enable parcial PCI Multifunction assignment with managed=true. That work will be scrapped after discussions with Laine in [1], but these cleanups kind of make sense on their own, so here they are. [1] https://www.redhat.com/archives/libvir-list/2019-July/msg01175.html *** BLURB HERE *** Daniel Henrique Barboza (3): virhostdev: introduce virHostdevResetAllPCIDevices virhostdev: remove virHostdevReattachPCIDevice virhostdev: introduce virHostdevReattachAllPCIDevices src/util/virhostdev.c | 148 +++++++++++++++++------------------------- src/util/virpci.c | 14 ++++ 2 files changed, 75 insertions(+), 87 deletions(-) -- 2.21.0

2 6

[libvirt] [PATCH v2 0/2] News for snapshot validation and vtpm encryption
by Han Han 31 Jul '19

31 Jul '19

v1: https://www.redhat.com/archives/libvir-list/2019-July/msg01897.html Han Han (2): news: Allow xml validation for snapshot creation news: Support encrypted soft tpm docs/news.xml | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) -- 2.20.1

3 6

[libvirt] [PATCH] backup: Add news entry for checkpoints
by Eric Blake 31 Jul '19

31 Jul '19

Checkpoints are definitely a news-worthy addition, even if the virDomainBackup API is not going to make it until a later release. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- docs/news.xml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/news.xml b/docs/news.xml index 1134309ec2..c7ac960523 100644 --- a/docs/news.xml +++ b/docs/news.xml @@ -63,6 +63,17 @@ <code>bochs-display</code> device that was added in qemu version 3.0. </description> </change> + <change> + <summary> + api: new virDomainCheckpoint APIs + </summary> + <description> + Introduce several new APIs for creating and managing + checkpoints in the test and qemu drivers (the latter + requires qcow2 images). Checkpoints serve as a way to tell + which portions of a disk have changed since a point in time. + </description> + </change> </section> <section title="Removed features"> <change> -- 2.20.1

2 1

[libvirt] [PATCH 0/6] Relax PCI domain limitation
by Michal Privoznik 31 Jul '19

31 Jul '19

As reported here [1], a PCI domain can be just any number. Well, not in case of traditional 32-bit PCI, bit in case of PCI-X (64-bit PCI; not to be confused with PCI Express), the PCI domain number can really be just anything. I wanted to download the specification from PCI SIG web but it's paid (why on earth would somebody create a standard and then make it not widely available is beyond me). 1: https://www.redhat.com/archives/libvir-list/2019-July/msg01732.html Michal Prívozník (6): qemuBuildPCIHostdevDevStr: Always format PCI domain onto cmd line virPCIDeviceNew: Prefer VIR_RETURN_PTR virPCIDevice: Make @name dynamically allocated lib: Unify PCI address formatting lib: Format PCI address differently virpci: Allow greater PCI domain value in virPCIDeviceAddressIsValid docs/schemas/basictypes.rng | 2 +- src/conf/device_conf.c | 4 +- src/conf/domain_addr.c | 10 ++-- src/conf/domain_audit.c | 3 +- src/conf/domain_conf.c | 12 ++--- src/conf/node_device_conf.c | 12 ++--- src/libxl/libxl_driver.c | 14 +++-- src/qemu/qemu_command.c | 9 ++-- src/qemu/qemu_hotplug.c | 5 +- src/util/virpci.c | 52 +++++++++---------- src/util/virpci.h | 2 + .../hostdev-pci-address-device.args | 2 +- .../qemuxml2argvdata/hostdev-pci-address.args | 2 +- .../hostdev-vfio-zpci-autogenerate.args | 2 +- .../hostdev-vfio-zpci-boundaries.args | 2 +- tests/qemuxml2argvdata/hostdev-vfio-zpci.args | 2 +- tests/qemuxml2argvdata/hostdev-vfio.args | 2 +- .../net-hostdev-bootorder.args | 3 +- tests/qemuxml2argvdata/net-hostdev-vfio.args | 2 +- tests/qemuxml2argvdata/net-hostdev.args | 2 +- tests/qemuxml2argvdata/pci-domain-invalid.xml | 2 +- tests/qemuxml2argvdata/pci-rom.args | 4 +- tools/virsh-domain.c | 4 +- 23 files changed, 81 insertions(+), 73 deletions(-) -- 2.21.0

3 11

[libvirt] [PATCH] gitdm: Add some more companies
by Andrea Bolognani 31 Jul '19

31 Jul '19

Employees from these companies have made contributions to libvirt over the past few releases. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- Pushed as trivial. docs/gitdm/companies/others | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/gitdm/companies/others b/docs/gitdm/companies/others index 440397df3d..db2c0061bd 100644 --- a/docs/gitdm/companies/others +++ b/docs/gitdm/companies/others @@ -14,6 +14,7 @@ citrix.com Citrix cloudwatt.com Cloudwatt codethink.co.uk Codethink cumulusnetworks.com Cumulus Networks +dataductus.se Data Ductus datagravity.com DataGravity dell.com Dell diateam.net DIATEAM @@ -67,6 +68,7 @@ otb.bg Open Technologies Bulgaria outscale.com OUTSCALE parallels.com Parallels petalogix.com PetaLogix +quobyte.com Quobyte ravellosystems.com Ravello Systems samsung.com Samsung sde.cz SDE @@ -89,6 +91,7 @@ virtualopensystems.com Virtual Open Systems websense.com Websense wiktel.com Wikstrom Telephone Company windriver.com Wind River +winhong.com Winhong xmission.com XMission xs4all.nl XS4ALL yadro.com YADRO -- 2.21.0

1 0