July 2019 - Devel - libvirt List Archives

[libvirt] [PATCH v10 00/19] Checkpoint APIs (incremental backup saga)
by Eric Blake 29 Jul '19

29 Jul '19

This is a subset of v10 of incremental backup, fixing Peter's review comments from v9 (and even some from v8 that I had missed earlier). There's still a lot more rebasing churn to resolve in the backup portion of the series before I can post a full backup-v10 label. There's still a decision to make: do we want this series in 5.6 (possibly with the addition of just patch 2/10 at [1] to introduce the backup API to make it possible for downstream to backport features without bumping .so)? This series has checkpoint support for test and qemu drivers, and I think is probably clean enough to finally satisfy everything Peter has been pointing out; but while the backup API itself seems reasonable, the qemu implementation will likely miss 5.6 (as that half of the v9 posting was further behind, and still has a lot of rebase churn to resolve). Or do we delay the checkpoint API to 5.7, to only go in with backup API? There's also still an outstanding question of whether the backup API needs a tweak to use 'const char *' instead of 'int' for the job identifier, given that Peter has a proposal for overhauling the representation of libvirt jobs. [1] https://www.redhat.com/archives/libvir-list/2019-July/msg00330.html Here's a comparison of these patches to v9: - rebase on top of master - use more VIR_AUTOFREE and friends (but probably still missed some spots where it would be worthwhile) - split several patches - add in a new patch to ensure the node names are learned at domain startup (but I didn't play enough with hotplug to see if that also has problems); necessary in order to... - get rid of detecting node names in checkpoint code - completely got rid of the notion of a current checkpoint in the public API (the internal code still has one, but solely to track which checkpoint becomes the parent of a new one to maintain a linear chain; when loading checkpoints on libvirtd restart, there should be at most one leaf node which becomes the internal current node) - improved code for handling partial checkpoints (the code now correctly handles: c1: disks A, B; c2: disks B, C; c3: disks A, B) 001/19:[down] 'snapshot: Don't leak moment obj list metaroot to callers' 002/19:[down] 'snapshot: Saner error message for duplicate create' 003/19:[down] 'snapshot: Documentation and comment improvements' 004/19:[down] 'backup: qemu: Detect node names at domain startup' 005/19:[0101] [FC] 'backup: Document new XML for checkpoints' 006/19:[0031] [FC] 'backup: Introduce virDomainCheckpoint APIs' 007/19:[0051] [FC] 'backup: Document nuances between different state capture APIs' 008/19:[0189] [FC] 'backup: Parse and output checkpoint XML' 009/19:[0066] [FC] 'backup: Allow for lists of checkpoint objects' 010/19:[----] [--] 'backup: Add new domain:checkpoint access control' 011/19:[----] [--] 'backup: Implement checkpoint APIs for remote driver' 012/19:[0161] [FC] 'backup: Implement virsh support for checkpoints' 013/19:[0047] [FC] 'backup: test: Implement metadata tracking for checkpoint APIs' 014/19:[0044] [FC] 'backup: Add virsh-checkpoints test' 015/19:[down] 'backup: qemu: Add directory for tracking checkpoints' 016/19:[0193] [FC] 'backup: qemu: Implement metadata tracking for checkpoint APIs' 017/19:[down] 'backup: qemu: Add helper API for looking up node name' 018/19:[0119] [FC] 'backup: Wire up qemu checkpoint commands over QMP' 019/19:[down] 'backup: Prevent snapshots and checkpoints at same time' Eric Blake (19): snapshot: Don't leak moment obj list metaroot to callers snapshot: Saner error message for duplicate create snapshot: Documentation and comment improvements backup: qemu: Detect node names at domain startup backup: Document new XML for checkpoints backup: Introduce virDomainCheckpoint APIs backup: Document nuances between different state capture APIs backup: Parse and output checkpoint XML backup: Allow for lists of checkpoint objects backup: Add new domain:checkpoint access control backup: Implement checkpoint APIs for remote driver backup: Implement virsh support for checkpoints backup: test: Implement metadata tracking for checkpoint APIs backup: Add virsh-checkpoints test backup: qemu: Add directory for tracking checkpoints backup: qemu: Implement metadata tracking for checkpoint APIs backup: qemu: Add helper API for looking up node name backup: Wire up qemu checkpoint commands over QMP backup: Prevent snapshots and checkpoints at same time include/libvirt/libvirt-domain-checkpoint.h | 137 ++ include/libvirt/libvirt-domain.h | 6 + include/libvirt/libvirt.h | 5 +- src/access/viraccessperm.h | 6 + src/conf/checkpoint_conf.h | 91 ++ src/conf/domain_conf.h | 2 + src/conf/virconftypes.h | 9 + src/conf/virdomaincheckpointobjlist.h | 72 + src/conf/virdomainmomentobjlist.h | 10 +- src/conf/virdomainobjlist.h | 7 +- src/conf/virdomainsnapshotobjlist.h | 2 + src/driver-hypervisor.h | 38 + src/qemu/qemu_conf.h | 2 + src/qemu/qemu_domain.h | 18 + tools/virsh-checkpoint.h | 26 + tools/virsh-completer.h | 4 + tools/virsh-util.h | 3 + tools/virsh.h | 1 + docs/Makefile.am | 3 + docs/apibuild.py | 2 + docs/docs.html.in | 4 +- docs/format.html.in | 1 + docs/formatcheckpoint.html.in | 196 +++ docs/formatsnapshot.html.in | 14 +- docs/index.html.in | 3 +- docs/kbase.html.in | 5 + docs/kbase/domainstatecapture.html.in | 303 +++++ docs/schemas/domaincheckpoint.rng | 94 ++ libvirt.spec.in | 2 + mingw-libvirt.spec.in | 4 + po/POTFILES | 3 + src/Makefile.am | 2 + src/access/viraccessperm.c | 3 +- src/conf/Makefile.inc.am | 4 + src/conf/checkpoint_conf.c | 597 +++++++++ src/conf/domain_conf.c | 6 + src/conf/virdomaincheckpointobjlist.c | 243 ++++ src/conf/virdomainmomentobjlist.c | 55 +- src/conf/virdomainobjlist.c | 11 + src/conf/virdomainsnapshotobjlist.c | 9 + src/libvirt-domain-checkpoint.c | 567 ++++++++ src/libvirt-domain.c | 19 +- src/libvirt_private.syms | 29 +- src/libvirt_public.syms | 16 + src/qemu/qemu_conf.c | 5 + src/qemu/qemu_domain.c | 190 +++ src/qemu/qemu_driver.c | 731 ++++++++++- src/qemu/qemu_process.c | 5 + src/remote/remote_daemon_dispatch.c | 20 + src/remote/remote_driver.c | 24 +- src/remote/remote_protocol.x | 123 +- src/remote_protocol-structs | 69 + src/rpc/gendispatch.pl | 32 +- src/test/test_driver.c | 386 +++++- tests/Makefile.am | 12 +- .../disk-default.xml | 7 + .../disk-invalid.xml | 8 + tests/qemudomaincheckpointxml2xmlin/empty.xml | 1 + .../name-invalid.xml | 4 + .../qemudomaincheckpointxml2xmlin/sample.xml | 7 + tests/qemudomaincheckpointxml2xmlin/size.xml | 4 + .../disk-default.xml | 11 + .../qemudomaincheckpointxml2xmlout/empty.xml | 7 + .../internal-active-invalid.xml | 53 + .../internal-inactive-invalid.xml | 53 + .../redefine.xml | 63 + .../qemudomaincheckpointxml2xmlout/sample.xml | 12 + tests/qemudomaincheckpointxml2xmlout/size.xml | 11 + tests/qemudomaincheckpointxml2xmltest.c | 213 +++ tests/virschematest.c | 2 + tests/virsh-checkpoint | 174 +++ tests/virsh-snapshot | 15 +- tools/Makefile.am | 1 + tools/virsh-checkpoint.c | 1153 +++++++++++++++++ tools/virsh-completer.c | 51 + tools/virsh-domain-monitor.c | 23 + tools/virsh-domain.c | 7 + tools/virsh-util.c | 11 + tools/virsh.c | 2 + tools/virsh.pod | 201 ++- 80 files changed, 6262 insertions(+), 63 deletions(-) create mode 100644 include/libvirt/libvirt-domain-checkpoint.h create mode 100644 src/conf/checkpoint_conf.h create mode 100644 src/conf/virdomaincheckpointobjlist.h create mode 100644 tools/virsh-checkpoint.h create mode 100644 docs/formatcheckpoint.html.in create mode 100644 docs/kbase/domainstatecapture.html.in create mode 100644 docs/schemas/domaincheckpoint.rng create mode 100644 src/conf/checkpoint_conf.c create mode 100644 src/conf/virdomaincheckpointobjlist.c create mode 100644 src/libvirt-domain-checkpoint.c create mode 100644 tests/qemudomaincheckpointxml2xmlin/disk-default.xml create mode 100644 tests/qemudomaincheckpointxml2xmlin/disk-invalid.xml create mode 100644 tests/qemudomaincheckpointxml2xmlin/empty.xml create mode 100644 tests/qemudomaincheckpointxml2xmlin/name-invalid.xml create mode 100644 tests/qemudomaincheckpointxml2xmlin/sample.xml create mode 100644 tests/qemudomaincheckpointxml2xmlin/size.xml create mode 100644 tests/qemudomaincheckpointxml2xmlout/disk-default.xml create mode 100644 tests/qemudomaincheckpointxml2xmlout/empty.xml create mode 100644 tests/qemudomaincheckpointxml2xmlout/internal-active-invalid.xml create mode 100644 tests/qemudomaincheckpointxml2xmlout/internal-inactive-invalid.xml create mode 100644 tests/qemudomaincheckpointxml2xmlout/redefine.xml create mode 100644 tests/qemudomaincheckpointxml2xmlout/sample.xml create mode 100644 tests/qemudomaincheckpointxml2xmlout/size.xml create mode 100644 tests/qemudomaincheckpointxml2xmltest.c create mode 100755 tests/virsh-checkpoint create mode 100644 tools/virsh-checkpoint.c -- 2.20.1

3 47

[libvirt] [PATCH] commandtest: Remove commandhelper.log in test27
by Michal Privoznik 29 Jul '19

29 Jul '19

The recently added test27 spawns commandhelper. This is fine, except, one of the things that commandhelper does is it records arguments it was spawn with into commandhelper.log. Other test cases then use checkoutput() to compare the arguments against the expected ones and also unlink() the log file. However, test27() is not doing that and thus it leaves the file behind. This breaks distcheck. Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com> --- Another approach would be to explicitly just unlink() the file somewhere in the cleanup label. But I figured, comparing the args and unlinking the file afterwards is more robust check IMO. tests/commanddata/test27.log | 20 ++++++++++++++++++++ tests/commandtest.c | 3 +++ 2 files changed, 23 insertions(+) create mode 100644 tests/commanddata/test27.log diff --git a/tests/commanddata/test27.log b/tests/commanddata/test27.log new file mode 100644 index 0000000000..40b2627c07 --- /dev/null +++ b/tests/commanddata/test27.log @@ -0,0 +1,20 @@ +ARG:--readfd +ARG:5 +ARG:--readfd +ARG:7 +ENV:DISPLAY=:0.0 +ENV:HOME=/home/test +ENV:HOSTNAME=test +ENV:LANG=C +ENV:LOGNAME=test +ENV:PATH=/usr/bin:/bin +ENV:TMPDIR=/tmp +ENV:USER=test +FD:0 +FD:1 +FD:2 +FD:5 +FD:7 +DAEMON:no +CWD:/tmp +UMASK:0022 diff --git a/tests/commandtest.c b/tests/commandtest.c index dfd15a2079..d7ab588969 100644 --- a/tests/commandtest.c +++ b/tests/commandtest.c @@ -1231,6 +1231,9 @@ static int test27(const void *unused ATTRIBUTE_UNUSED) goto cleanup; } + if (checkoutput("test27", NULL) < 0) + goto cleanup; + ret = 0; cleanup: -- 2.21.0

2 1

[libvirt] [PATCH v2 0/2] test_driver: implement virDomainGetCPUStats
by Ilias Stamatis 29 Jul '19

29 Jul '19

Changes since v1: * Removed failing test cases * Total CPUTIME now is the sum of the individual CPUs Ilias Stamatis (2): tests: virsh-optparse: remove no longer valid cpu-stats test cases test_driver: implement virDomainGetCPUStats src/test/test_driver.c | 132 +++++++++++++++++++++++++++++++++++++++++ tests/virsh-optparse | 20 ------- 2 files changed, 132 insertions(+), 20 deletions(-) -- 2.22.0

2 6

[libvirt] Some windows hyperv features cannot be enabled after the patch 63acb7bfd56f117309e4fcaf438639d4d7bc7dcb
by Changlimin 29 Jul '19

29 Jul '19

When using with QEMU v4.1.0-rc0<http://10.114.102.26/cgit/qemu/tag/?h=v4.1.0-rc0> or newer, the patch https://libvirt.org/git/?p=libvirt.git;a=commit;h=63acb7bfd56f117309e4fcaf4… cause some windows hyperv features cannot be enabled. # virsh start win10 error: Failed to start domain win10 error: unsupported configuration: host doesn't support hyperv 'vpindex' feature <features> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='4096'/> <vpindex state='on'/> <runtime state='on'/> <synic state='on'/> <reset state='on'/> <frequencies state='on'/> <reenlightenment state='on'/> <tlbflush state='on'/> <ipi state='on'/> <evmcs state='on'/> </hyperv>

2 1

[libvirt] [PATCH 0/5] Fix hyperv and kvm features with QEMU 4.1
by Jiri Denemark 29 Jul '19

29 Jul '19

Originally the names of the hyperv and kvm CPU features were only used internally for looking up their CPUID bits. But with QEMU 4.1 we check which features were enabled or disabled by a freshly started QEMU process using their names rather than their CPUID bits (mostly because of MSR features). Thus we need to change our made up internal names into the actual names used by QEMU. Otherwise libvirt would mistakenly report the features as unavailable and refuse to start any domain using them with QEMU 4.1. Reported-by: Vitaly Kuznetsov <vkuznets(a)redhat.com> Jiri Denemark (5): qemu: Fix hyperv features with QEMU 4.1 qemu: Prefer dashes for hyperv features cpu: Drop KVM_ from hyperv feature macros cpu: Drop unused KVM features qemu: Fix KVM features with QEMU 4.1 src/cpu/cpu_x86.c | 77 +++++++------------ src/cpu/cpu_x86_data.h | 38 ++++----- src/qemu/qemu_command.c | 19 +++-- src/qemu/qemu_process.c | 2 +- .../clock-timer-hyperv-rtc.args | 2 +- tests/qemuxml2argvdata/hyperv-panic.args | 2 +- tests/qemuxml2argvdata/hyperv.args | 4 +- tests/qemuxml2argvdata/panic-double.args | 2 +- 8 files changed, 61 insertions(+), 85 deletions(-) -- 2.22.0

3 7

[libvirt] [PATCH 0/6] Add support for Direct Mode for Hyper-V Synthetic timers
by Vitaly Kuznetsov 29 Jul '19

29 Jul '19

QEMU-4.1 will bring us Direct Mode for Hyper-V Synthetic timers support, we need to support it in libvirt too. As this is not a new enlightenment but rather an enhancement of an existing one ('stimer'), support it in <stimer state='on'> <direct state='on'/> </stimer> form. Backwards compatibility is (hopefully) preserved. Because of an existing imcompatibility between libvirt-5.5+ and QEMU-4.1-git (see https://www.redhat.com/archives/libvir-list/2019-July/msg01435.html) I had to backport hv-stimer-direct to QEMU-4.0 to test this. If someone wants to do it too here is a list of QEMU patches needed: 128531d9e1 i386/kvm: add support for Direct Mode for Hyper-V synthetic timers 8caba36db5 i386/kvm: hv-evmcs requires hv-vapic bd59fbdf4f i386/kvm: hv-tlbflush/ipi require hv-vpindex c686193072 i386/kvm: hv-stimer requires hv-time and hv-synic e48ddcc6ce i386/kvm: implement 'hv-passthrough' mode fb19f72b77 i386/kvm: document existing Hyper-V enlightenments 2344d22e50 i386/kvm: move Hyper-V CPUID filling to hyperv_handle_properties() 6760bd2002 i386/kvm: add support for KVM_GET_SUPPORTED_HV_CPUID 2d384d7c83 i386/kvm: convert hyperv enlightenments properties from bools to bits Vitaly Kuznetsov (6): docs: formatdomain: add synic flag to the example xml docs: formatdomain: fix 'SynIC' spelling docs: formatdomain: move 'msrs' out of Hyper-V Enlightenments conf: add support for Direct Mode for Hyper-V Synthetic timers qemu: add support for Direct Mode for Hyper-V Synthetic timers news: mention Direct Mode for Hyper-V Synthetic timers support docs/formatdomain.html.in | 13 ++- docs/news.xml | 9 ++ docs/schemas/domaincommon.rng | 16 +++- src/conf/domain_conf.c | 138 +++++++++++++++++++++++++--- src/conf/domain_conf.h | 8 ++ src/cpu/cpu_x86.c | 51 +++++----- src/cpu/cpu_x86_data.h | 2 + src/libvirt_private.syms | 2 + src/qemu/qemu_command.c | 22 ++++- src/qemu/qemu_process.c | 39 +++++++- tests/qemuxml2argvdata/hyperv.args | 4 +- tests/qemuxml2argvdata/hyperv.xml | 4 +- tests/qemuxml2xmloutdata/hyperv.xml | 4 +- 13 files changed, 263 insertions(+), 49 deletions(-) -- 2.20.1

2 12

[libvirt] [PATCH] qemumonitorjsontest: Don't leak 'query-jobs' info
by Michal Privoznik 29 Jul '19

29 Jul '19

The returned array of qemuMonitorJobInfo structs must be freed. 164 (16 direct, 148 indirect) bytes in 1 blocks are definitely lost in loss record 64 of 84 at 0x4A3568B: realloc (vg_replace_malloc.c:826) by 0x4D888BD: virReallocN (viralloc.c:244) by 0x4D889B3: virExpandN (viralloc.c:293) by 0x4D88C87: virInsertElementsN (viralloc.c:435) by 0x214004: qemuMonitorJSONGetJobInfo (qemu_monitor_json.c:9185) by 0x148B3F: testQueryJobs (qemumonitorjsontest.c:2979) by 0x14C192: virTestRun (testutils.c:174) by 0x14BF36: mymain (qemumonitorjsontest.c:3286) by 0x14E256: virTestMain (testutils.c:1096) by 0x14BFD9: main (qemumonitorjsontest.c:3298) Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com> --- tests/qemumonitorjsontest.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/qemumonitorjsontest.c b/tests/qemumonitorjsontest.c index bf89f49aca..522ff5a3d5 100644 --- a/tests/qemumonitorjsontest.c +++ b/tests/qemumonitorjsontest.c @@ -2996,6 +2996,9 @@ testQueryJobs(const void *opaque) ret = 0; cleanup: + for (i = 0; i < njobs; i++) + qemuMonitorJobInfoFree(jobs[i]); + VIR_FREE(jobs); qemuMonitorTestFree(test); return ret; } -- 2.21.0

2 1

[libvirt] UID/GID during kvm/qemu migrate
by Stephan von Krawczynski 29 Jul '19

29 Jul '19

Hello, is there some immanent code in libvirt that forces UID/GID of the libvirt standard user to be the same on two boxes migrating qemu vms against each other? The migration itself uses root obviously (password is requested). But if a vm xml does not contain any definition regarding UID/GID what else could prevent this from working? I believe I ran into such a problem trying to migrate and ending up in an error, a vm still working on original host but its fs (netfs pool (nfs/raw)) being switched to read-only... -- Regards, Stephan

1 0

[libvirt] [PATCH 0/9] qemu: Support block pull/commit jobs with blockdev (blockdev-add saga)
by Peter Krempa 29 Jul '19

29 Jul '19

One of the final chapters of the saga. At the end we finally support first blockjobs with blockdev. This series depends on some of the previous postings so for convenience it can be fetched at: git fetch https://gitlab.com/pipo.sk/libvirt.git blockjobs-send-partial The above branch also contains patches which enable blockdev support, which is ideal for testing. Peter Krempa (9): qemu: blockjob: Use proper value when setting disk's READY state tests: qemustatusxml: Add few disks with backing chains for blockdev job testing qemu: blockjob: Document qemuBlockJobRegister qemu: blockjob: Register disk->mirror with a job only when required qemu: blockjob: Split out update of persistent XML disk's source on mirror jobs qemu: blockjob: Esure that persistent definition source does not copy irrelevant data qemu: blockjob: Ensure that config disk source is identical when modifying it qemu: Add -blockdev support for block pull job qemu: Add -blockdev support for block commit job src/qemu/qemu_blockjob.c | 525 +++++++++++++++++- src/qemu/qemu_blockjob.h | 40 ++ src/qemu/qemu_domain.c | 123 +++- src/qemu/qemu_driver.c | 83 ++- .../blockjob-blockdev-in.xml | 285 +++++++++- 5 files changed, 1001 insertions(+), 55 deletions(-) -- 2.21.0

3 21

[libvirt] virsh command to list CPU, Memory and Storage of all the running VM's
by Kaushal Shriyan 29 Jul '19

29 Jul '19

Hi, Is there a way to find out total CPU's, Memory and Storage of all the running VM's using virsh command? For example virsh list --all list out all VM's 11 dockerregistry01 running 12 gitlab running Any help will be highly appreciated. Thanks in Advance. Best Regards, Kaushal

2 1