Re: [libvirt] [Qemu-devel] Virtio-net control queue handling
by Stefan Hajnoczi
On Tue, Feb 26, 2019 at 02:13:43PM +0000, Nikhil Agarwal wrote:
> Is there any option in QEMU to offload virtio device control queue handling to backend instead of deciphering and passing to libvirt via QMP? if not, is there any interface in libvirt which passes on these message to application or i directly use QMP interface from QEMU?
I'm not sure I understand your question. It could be because of
terminology:
"virtio device control queue" == virtio-net control virtqueue?
"backend" == vhost-user-net device backend?
"message" == QMP event?
I have CCed Jason Wang, QEMU network subsystem maintainer, and the
libvirt mailing list.
Stefan
5 years, 7 months
[libvirt] [PATCH] snapshot: Improve message for VIR_ERR_INVALID_DOMAIN_SNAPSHOT
by Eric Blake
For consistency with other error messages, and the fact that
the object is always called a virDomainSnapshot rather than
a mere virSnapshot, include the word "domain" in the error
message.
Suggested-by: John Ferlan <jferlan(a)redhat.com>
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
src/util/virerror.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/util/virerror.c b/src/util/virerror.c
index 9e3975fe76..05e535d859 100644
--- a/src/util/virerror.c
+++ b/src/util/virerror.c
@@ -1123,8 +1123,8 @@ const virErrorMsgTuple virErrorMsgStrings[VIR_ERR_NUMBER_LAST] = {
N_("Hook script execution failed"),
N_("Hook script execution failed: %s") },
[VIR_ERR_INVALID_DOMAIN_SNAPSHOT] = {
- N_("Invalid snapshot"),
- N_("Invalid snapshot: %s") },
+ N_("Invalid domain snapshot"),
+ N_("Invalid domain snapshot: %s") },
[VIR_ERR_NO_DOMAIN_SNAPSHOT] = {
N_("Domain snapshot not found"),
N_("Domain snapshot not found: %s") },
--
2.20.1
5 years, 7 months
[libvirt] [PATCH v2 00/11] bulk snapshot list/redefine (incremental backup saga)
by Eric Blake
While looking at my work on incremental backups, Nir raised a good
point: if we want to recreate a set of known checkpoints on one
machine that will be taking over a domain from another machine,
my initial proposal required making multiple API calls to list the
XML for each checkpoint on the source, and then more API calls to
redefine each checkpoint on the destination; it also had the drawback
that the list has to be presented in topological order (the API won't
let you define a child checkpoint if the parent is not defined first).
He asked if I could instead add bulk APIs, for getting the XML for
all checkpoints at once, and then for redefining all checkpoints at
once.
Since my checkpoint code borrows heavily from concepts in the snapshot
code, I chose to tackle the problem by starting with this series, which
does the same thing for snapshots as what I plan to do for checkpoints.
That is, since this patch series adds virDomainGetXMLDesc(,
VIR_DOMAIN_XML_SNAPSHOTS) and virDomainSnapshotCreateXML(,
VIR_DOMAIN_SNAPSHOT_CREATE_REDEFINE_LIST), the checkpoint series
will add virDOmainGetXMLDesc(, VIR_DOMAIN_XML_CHECKPOINTS) and
virDomainCheckpointCreateXML(, VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE_LIST)
with very similar code.
Also available at:
https://repo.or.cz/libvirt/ericb.git/shortlog/refs/tags/snapshot-bulk-v2
Depends on:
https://www.redhat.com/archives/libvir-list/2019-February/msg01263.html
(virsh: Treat \n like ; in batch mode)
https://www.redhat.com/archives/libvir-list/2019-February/msg01343.html
(fix snapshot --redefine bugs)
Since v1:
Finish implementing the redefine code
Split the patches slightly differently: all snapshot_conf code first,
then just the public API changes, then virsh wrappers, then per-driver
implementations for test and qemu.
Rebase on top of bug fix and virsh feature addition found in the meantime.
001/11:[----] [--] 'domain: Document VIR_DOMAIN_XML_MIGRATABLE'
002/11:[----] [-C] 'snapshot: Give virDomainSnapshotDefFormat its own flags'
003/11:[0012] [FC] 'snapshot: Refactor virDomainSnapshotDefFormat'
004/11:[----] [-C] 'snapshot: Add virDomainSnapshotObjListFormat'
005/11:[----] [-C] 'domain: Expand virDomainDefFormatInternal with snapshots'
006/11:[down] 'snapshot: Add virDomainSnapshotDefParseList'
007/11:[0041] [FC] 'domain: Add VIR_DOMAIN_XML_SNAPSHOTS flag'
008/11:[0013] [FC] 'snapshot: Add VIR_DOMAIN_SNAPSHOT_CREATE_REDEFINE_LIST flag'
009/11:[down] 'virsh: Expose bulk snapshot dumpxml/redefine'
010/11:[down] 'test: Implement bulk snapshot operations'
011/11:[down] 'qemu: Implement bulk snapshot operations'
Eric Blake (11):
domain: Document VIR_DOMAIN_XML_MIGRATABLE
snapshot: Give virDomainSnapshotDefFormat its own flags
snapshot: Refactor virDomainSnapshotDefFormat
snapshot: Add virDomainSnapshotObjListFormat
domain: Expand virDomainDefFormatInternal with snapshots
snapshot: Add virDomainSnapshotDefParseList
domain: Add VIR_DOMAIN_XML_SNAPSHOTS flag
snapshot: Add VIR_DOMAIN_SNAPSHOT_CREATE_REDEFINE_LIST flag
virsh: Expose bulk snapshot dumpxml/redefine
test: Implement bulk snapshot operations
qemu: Implement bulk snapshot operations
include/libvirt/libvirt-domain-snapshot.h | 3 +
include/libvirt/libvirt-domain.h | 1 +
src/conf/domain_conf.h | 8 +
src/conf/snapshot_conf.h | 25 +-
src/qemu/qemu_domain.h | 2 +-
src/conf/domain_conf.c | 71 +++++-
src/conf/snapshot_conf.c | 297 ++++++++++++++++++----
src/esx/esx_driver.c | 1 -
src/libvirt-domain-snapshot.c | 21 +-
src/libvirt-domain.c | 14 +
src/libvirt_private.syms | 3 +
src/network/bridge_driver.c | 3 +-
src/qemu/qemu_domain.c | 36 ++-
src/qemu/qemu_driver.c | 67 ++++-
src/test/test_driver.c | 28 +-
src/vbox/vbox_common.c | 5 +-
src/vz/vz_driver.c | 3 +-
tests/domainsnapshotxml2xmltest.c | 16 +-
tools/virsh-domain.c | 7 +
tools/virsh-snapshot.c | 14 +
20 files changed, 523 insertions(+), 102 deletions(-)
--
2.20.1
5 years, 7 months
[libvirt] [PATCH v3 00/10] dirty-bitmaps: deprecate @status field
by John Snow
The current internal meanings of "locked", "user_locked",
"qmp_locked", "frozen", "enabled", and "disabled" are all
a little muddled.
Deprecate the @status field in favor of two new booleans
that carry very specific meanings. Then, rename and rework
some of the internal semantics to help make the API a bit
more clear and easier to read.
Well, in my opinion.
Based on my current bitmaps branch.
V3:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/10:[0002] [FC] 'block/dirty-bitmap: add recording and busy properties'
002/10:[0002] [FC] 'block/dirty-bitmaps: rename frozen predicate helper'
003/10:[down] 'block/dirty-bitmap: remove set/reset assertions against enabled bit'
004/10:[0006] [FC] 'block/dirty-bitmap: change semantics of enabled predicate'
005/10:[down] 'nbd: change error checking order for bitmaps'
006/10:[0002] [FC] 'block/dirty-bitmap: explicitly lock bitmaps with successors'
007/10:[0011] [FC] 'block/dirty-bitmaps: unify qmp_locked and user_locked calls'
008/10:[0002] [FC] 'block/dirty-bitmaps: move comment block'
009/10:[down] 'blockdev: remove unused paio parameter documentation'
010/10:[down] 'iotests: add busy/recording bit test to 124'
V3: (Following Vladimir's feedback)
001: Changed texi phrasing, parameter --> field
002: Commit message adjustment
comment edit: "not locked" --> "not user_locked"
003: pulled forward out of patch 004
004: Commit message rewrite
create_successor and abdicate doc adjustments
005: New.
006: Change successor doc comment
Commit message adjustment
007: BdrvDirtyBitmap struct comment adjustments
Comment change to create_successor (lock --> busy)
Incidental changes from 004's changes
008: Grammar fix (Eric)
009: New, trivial, unrelated fix tagging along.
010: iotest 124 added.
John Snow (10):
block/dirty-bitmap: add recording and busy properties
block/dirty-bitmaps: rename frozen predicate helper
block/dirty-bitmap: remove set/reset assertions against enabled bit
block/dirty-bitmap: change semantics of enabled predicate
nbd: change error checking order for bitmaps
block/dirty-bitmap: explicitly lock bitmaps with successors
block/dirty-bitmaps: unify qmp_locked and user_locked calls
block/dirty-bitmaps: move comment block
blockdev: remove unused paio parameter documentation
iotests: add busy/recording bit test to 124
block/dirty-bitmap.c | 111 ++++++++++++++++++---------------
blockdev.c | 19 +++---
include/block/dirty-bitmap.h | 7 +--
migration/block-dirty-bitmap.c | 8 +--
nbd/server.c | 14 ++---
qapi/block-core.json | 10 ++-
qemu-deprecated.texi | 6 ++
tests/qemu-iotests/124 | 110 ++++++++++++++++++++++++++++++++
tests/qemu-iotests/124.out | 4 +-
tests/qemu-iotests/236.out | 28 +++++++++
10 files changed, 239 insertions(+), 78 deletions(-)
--
2.17.2
5 years, 7 months
[libvirt] [PATCH] virsh: Treat \n like ; in batch mode
by Eric Blake
I wanted to do a demonstration with virsh batch mode, which
takes multiple commands all packed into a single argument:
$ virsh -c test:///default 'echo a; echo b;'
a
b
but that produced a really long line, so I tried to make it
more legible:
$ virsh -c test:///default '
echo a;
echo b;
'
error: unknown command: '
'
Let's be more like the shell, and treat unquoted newline as a
command separator just as we do for semicolon. In fact, with
that, I can even now mix styles:
$ virsh -c test:///default '
echo a; echo b
echo c
'
a
b
c
Fix the grammer in a nearby comment while at it.
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
tools/virsh.pod | 4 ++--
tools/virt-admin.pod | 4 ++--
tools/vsh.c | 6 +++---
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/tools/virsh.pod b/tools/virsh.pod
index 67edb57b14..2bf1ee77bb 100644
--- a/tools/virsh.pod
+++ b/tools/virsh.pod
@@ -40,8 +40,8 @@ as a name.
The B<virsh> program can be used either to run one I<COMMAND> by giving the
command and its arguments on the shell command line, or a I<COMMAND_STRING>
which is a single shell argument consisting of multiple I<COMMAND> actions
-and their arguments joined with whitespace, and separated by semicolons
-between commands. Within I<COMMAND_STRING>, virsh understands the
+and their arguments joined with whitespace, and separated by semicolons or
+newlines between commands. Within I<COMMAND_STRING>, virsh understands the
same single, double, and backslash escapes as the shell, although you must
add another layer of shell escaping in creating the single shell argument.
If no command is given in the command line, B<virsh> will then start a minimal
diff --git a/tools/virt-admin.pod b/tools/virt-admin.pod
index 3ddbbff934..b2c9a1db6d 100644
--- a/tools/virt-admin.pod
+++ b/tools/virt-admin.pod
@@ -23,8 +23,8 @@ Where I<command> is one of the commands listed below.
The B<virt-admin> program can be used either to run one I<COMMAND> by giving the
command and its arguments on the shell command line, or a I<COMMAND_STRING>
which is a single shell argument consisting of multiple I<COMMAND> actions
-and their arguments joined with whitespace, and separated by semicolons
-between commands. Within I<COMMAND_STRING>, virt-admin understands the
+and their arguments joined with whitespace, and separated by semicolons or
+newlines between commands. Within I<COMMAND_STRING>, virt-admin understands the
same single, double, and backslash escapes as the shell, although you must
add another layer of shell escaping in creating the single shell argument.
If no command is given in the command line, B<virt-admin> will then start a minimal
diff --git a/tools/vsh.c b/tools/vsh.c
index 610de4495b..3d9bec896b 100644
--- a/tools/vsh.c
+++ b/tools/vsh.c
@@ -1664,7 +1664,7 @@ vshCommandStringGetArg(vshControl *ctl, vshCommandParser *parser, char **res,
if (*p == '\0')
return VSH_TK_END;
- if (*p == ';') {
+ if (*p == ';' || *p == '\n') {
parser->pos = ++p; /* = \0 or begin of next command */
return VSH_TK_SUBCMD_END;
}
@@ -1672,7 +1672,7 @@ vshCommandStringGetArg(vshControl *ctl, vshCommandParser *parser, char **res,
while (*p) {
/* end of token is blank space or ';' */
if (!double_quote && !single_quote &&
- (*p == ' ' || *p == '\t' || *p == ';'))
+ (*p == ' ' || *p == '\t' || *p == ';' || *p == '\n'))
break;
if (!double_quote && *p == '\'') { /* single quote */
@@ -1681,7 +1681,7 @@ vshCommandStringGetArg(vshControl *ctl, vshCommandParser *parser, char **res,
continue;
} else if (!single_quote && *p == '\\') { /* escape */
/*
- * The same as the bash, a \ in "" is an escaper,
+ * The same as in shell, a \ in "" is an escaper,
* but a \ in '' is not an escaper.
*/
p++;
--
2.20.1
5 years, 7 months
[libvirt] [tck PATCH] Add script testing network bandwidth functionality
by Daniel P. Berrangé
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
lib/Sys/Virt/TCK/DomainBuilder.pm | 10 ++
lib/Sys/Virt/TCK/NetworkBuilder.pm | 20 ++++
scripts/networks/400-guest-bandwidth.t | 147 +++++++++++++++++++++++++
3 files changed, 177 insertions(+)
create mode 100644 scripts/networks/400-guest-bandwidth.t
diff --git a/lib/Sys/Virt/TCK/DomainBuilder.pm b/lib/Sys/Virt/TCK/DomainBuilder.pm
index 399534c..45336b5 100644
--- a/lib/Sys/Virt/TCK/DomainBuilder.pm
+++ b/lib/Sys/Virt/TCK/DomainBuilder.pm
@@ -466,6 +466,16 @@ sub as_xml {
$w->emptyTag("model",
type => $interface->{model});
}
+ if ($interface->{bandwidth}) {
+ $w->startTag("bandwidth");
+ if ($interface->{bandwidth}->{"in"}) {
+ $w->emptyTag("inbound", %{$interface->{bandwidth}->{"in"}});
+ }
+ if ($interface->{bandwidth}->{"out"}) {
+ $w->emptyTag("outbound", %{$interface->{bandwidth}->{"out"}});
+ }
+ $w->endTag("bandwidth");
+ }
if ($interface->{filterref}) {
$w->startTag("filterref",
filter => $interface->{filterref});
diff --git a/lib/Sys/Virt/TCK/NetworkBuilder.pm b/lib/Sys/Virt/TCK/NetworkBuilder.pm
index ad0cab8..446cc4f 100644
--- a/lib/Sys/Virt/TCK/NetworkBuilder.pm
+++ b/lib/Sys/Virt/TCK/NetworkBuilder.pm
@@ -87,6 +87,15 @@ sub ipaddr {
return $self;
}
+sub bandwidth {
+ my $self = shift;
+ my %bw = @_;
+
+ $self->{bandwidth} = \%bw;
+
+ return $self;
+}
+
sub dhcp_range {
my $self = shift;
my $start = shift;
@@ -114,6 +123,17 @@ sub as_xml {
$w->emptyTag("bridge", %{$self->{bridge}})
if $self->{bridge};
+ if ($self->{bandwidth}) {
+ $w->startTag("bandwidth");
+ if ($self->{bandwidth}->{"in"}) {
+ $w->emptyTag("inbound", %{$self->{bandwidth}->{"in"}});
+ }
+ if ($self->{bandwidth}->{"out"}) {
+ $w->emptyTag("outbound", %{$self->{bandwidth}->{"out"}});
+ }
+ $w->endTag("bandwidth");
+ }
+
if (exists $self->{forward}) {
$w->startTag("forward", %{$self->{forward}});
foreach (@{$self->{interfaces}}) {
diff --git a/scripts/networks/400-guest-bandwidth.t b/scripts/networks/400-guest-bandwidth.t
new file mode 100644
index 0000000..3a865fd
--- /dev/null
+++ b/scripts/networks/400-guest-bandwidth.t
@@ -0,0 +1,147 @@
+# -*- perl -*-
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; You can redistribute it and/or modify
+# it under the GNU General Public License as published by the Free
+# Software Foundation; either version 2, or (at your option) any
+# later version
+#
+# The file "LICENSE" distributed along with this file provides full
+# details of the terms and conditions
+#
+
+=pod
+
+=head1 NAME
+
+network/400-guest-bandwidth.t - guest with bandwidth limits
+
+=head1 DESCRIPTION
+
+This test case validates that a guest is connected a network
+can have bandwidth limits set
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More tests => 8;
+use Test::Exception;
+
+use Sys::Virt::TCK;
+
+my $tck = Sys::Virt::TCK->new();
+my $conn = eval { $tck->setup(); };
+BAIL_OUT "failed to setup test harness: $@" if $@;
+END { $tck->cleanup if $tck; }
+
+my %subnet = Sys::Virt::TCK->find_free_ipv4_subnet();
+
+SKIP: {
+ skip "No available IPv4 subnet", 4 unless defined $subnet{address};
+
+ my $netbuilder = Sys::Virt::TCK::NetworkBuilder->new(name => "tck");
+ $netbuilder->bridge("tck");
+ $netbuilder->ipaddr($subnet{address}, $subnet{netmask});
+ $netbuilder->dhcp_range($subnet{dhcpstart}, $subnet{dhcpend});
+ my $netxml = $netbuilder->as_xml();
+
+ diag "Creating a new transient network";
+ diag $netxml;
+ my $net;
+ ok_network(sub { $net = $conn->create_network($netxml) }, "created transient network object");
+
+ my $dombuilder = $tck->generic_domain(name => "tck");
+ $dombuilder->interface(type => "network",
+ source => "tck",
+ model => "virtio",
+ mac => "52:54:00:11:11:11",
+ bandwidth => {
+ in => {
+ average => 1000,
+ peak => 5000,
+ floor => 2000,
+ burst => 1024,
+ },
+ out => {
+ average => 128,
+ peak => 256,
+ burst => 256,
+ },
+ });
+ my $domxml = $dombuilder->as_xml();
+
+ diag "Creating a new transient domain";
+ diag $domxml;
+ my $dom;
+ ok_error(sub { $dom = $conn->create_domain($domxml) }, "Unsupported op requesting bandwidth",
+ Sys::Virt::Error::ERR_OPERATION_UNSUPPORTED);
+
+ diag "Destroying the transient network";
+ $net->destroy;
+
+ $netbuilder->bandwidth(
+ in => {
+ average => 1000,
+ peak => 5000,
+ burst => 1024,
+ },
+ out => {
+ average => 128,
+ peak => 256,
+ burst => 256,
+ });
+
+ $netxml = $netbuilder->as_xml();
+
+ diag "Creating a new transient network";
+ diag $netxml;
+ ok_network(sub { $net = $conn->create_network($netxml) }, "created transient network object");
+
+ ok_domain(sub { $dom = $conn->create_domain($domxml) }, "created transient domain object");
+
+ lives_ok(sub {
+ $dom->set_interface_parameters(
+ "52:54:00:11:11:11",
+ {
+ Sys::Virt::Domain::BANDWIDTH_IN_AVERAGE => 1000,
+ Sys::Virt::Domain::BANDWIDTH_IN_PEAK => 5000,
+ Sys::Virt::Domain::BANDWIDTH_IN_FLOOR => 4000,
+ Sys::Virt::Domain::BANDWIDTH_IN_BURST => 1024,
+ Sys::Virt::Domain::BANDWIDTH_OUT_AVERAGE => 128,
+ Sys::Virt::Domain::BANDWIDTH_OUT_PEAK => 256,
+ Sys::Virt::Domain::BANDWIDTH_OUT_BURST => 256,
+ })});
+
+ ok_error(sub {
+ $dom->set_interface_parameters(
+ "52:54:00:11:11:11",
+ {
+ Sys::Virt::Domain::BANDWIDTH_IN_AVERAGE => 1000,
+ Sys::Virt::Domain::BANDWIDTH_IN_PEAK => 5000,
+ Sys::Virt::Domain::BANDWIDTH_IN_FLOOR => 40000,
+ Sys::Virt::Domain::BANDWIDTH_IN_BURST => 1024,
+ Sys::Virt::Domain::BANDWIDTH_OUT_AVERAGE => 128,
+ Sys::Virt::Domain::BANDWIDTH_OUT_PEAK => 256,
+ Sys::Virt::Domain::BANDWIDTH_OUT_BURST => 256,
+ }) }, "Canot overcommit bandwidth",
+ Sys::Virt::Error::ERR_OPERATION_INVALID);
+
+ diag "Destroying the transient guest";
+ $dom->destroy;
+
+ diag "Checking that transient domain has gone away";
+ ok_error(sub { $conn->get_domain_by_name("tck") }, "NO_DOMAIN error raised from missing domain",
+ Sys::Virt::Error::ERR_NO_DOMAIN);
+
+ diag "Destroying the transient network";
+ $net->destroy;
+
+ diag "Checking that transient network has gone away";
+ ok_error(sub { $conn->get_network_by_name("tck") }, "NO_network error raised from missing network",
+ Sys::Virt::Error::ERR_NO_NETWORK);
+}
+
+# end
--
2.20.1
5 years, 7 months
[libvirt] [PATCH 0/5] Yet another set of VIR_AUTO-related patches
by Peter Krempa
Peter Krempa (5):
cfg: Add VIR_AUTO(UNREF|CLEAN|STRINGLIST) to
sc_requirere_attribute_cleanup_initialization
util: object: Reset pointer when unrefing object in virObjectAutoUnref
util: alloc: Clarify docs for VIR_DEFINE_AUTOCLEAN_FUNC
util: XML: Introduce automatic reset of XPath's current node
qemu: Use VIR_XPATH_NODE_AUTORESTORE when XPath context is modified
cfg.mk | 2 +-
src/libvirt_private.syms | 1 +
src/qemu/qemu_capabilities.c | 3 +--
src/qemu/qemu_domain.c | 6 ++----
src/qemu/qemu_migration_cookie.c | 9 +++------
src/util/viralloc.h | 4 ++--
src/util/virobject.c | 1 +
src/util/virxml.c | 10 ++++++++++
src/util/virxml.h | 22 ++++++++++++++++++++++
9 files changed, 43 insertions(+), 15 deletions(-)
--
2.20.1
5 years, 7 months
[libvirt] Libvirt upstream CI efforts
by Erik Skultety
Hi,
I'm starting this thread in order to continue with the ongoing efforts to
bring actual integration testing to libvirt. Currently, the status quo is that
we build libvirt (along with our unit test suite) using different OS-flavoured
VMs in ci.centos.org. Andrea put a tremendous amount of work to not only
automate the whole process of creating the VMs but also having a way for a
dev to re-create the same environment locally without jenkins by using the
lcitool.
#TL;DR (if you're from QEMU, no TLDR for you ;), there are questions to answer)
- we need to run functional tests upstream on ci.centos.org
-> pure VM testing environment (nested for migration) vs Docker images
- we need to host the upstream test suite somewhere
-> main libvirt.git repo vs libvirt-jenkins-ci.git vs new standalone repo
- what framework to use for the test suite
-> TCK vs avocado-vt vs plain avocado
#THE LONG STORY SHORT
As far as the functional test suite goes, there's an already existing
integration with the avocado-vt and a massive number of test cases at [1]
which is currently not used for upstream testing, primarily because of the huge
number of test cases (and also many unnecessary legacy test cases). An
alternative set of functional test cases is available as part of the
libvirt-tck framework [2]. The obvious question now is how can we build upon
any of this and introduce proper functional testing of upstream libvirt to our
jenkins environment at ci.centos.org, so I formulated the following discussion
points as I think these are crucial to sort out before we move on to the test
suite itself:
* Infrastructure/Storage requirements (need for hosting pre-build images?)
- one of the main goals we should strive for with upstream CI is that
every developer should be able to run the integration test suite on
their own machine (conveniently) prior to submitting their patchset to
the list
- we need a reproducible environment to ensure that we don't get different
results across different platforms (including ci.centos.org), therefore
we could provide pre-built images with environment already set up to run
the suite in an L1 guest.
- as for performing migration tests, we could utilize nested virt
- should we go this way, having some publicly accessible storage to host
all the pre-built images is a key problem to solve
-> an estimate of how much we're currently using: roughly 130G from
our 500G allocation at ci.centos.org to store 8 qcow2 images + 2
freebsd isos
-> we're also fairly generous with how much we allocate for a guest
image as most of the guests don't even use half of the 20G
allocation
-> considering sparsifying the pre-built images and compressing them
+ adding a ton of dependencies to run the suite, extending the
pool of distros by including ubuntu 16 + 18, 200-250G is IMHO
quite a generous estimate of our real need
-> we need to find a party willing to give us the estimated amount
of publicly accessible storage and consider whether we'd need any
funds for that
-> we'd have to also talk to other projects that have done a similar
thing about possible caveats related to hosting images, e.g.
bandwidth
-> as for ci.centos.org, it does provide publicly accessible folder
where projects can store artifacts (the documentation even
mentions VM images), there might a limit though [3]
- alternatively, we could use Docker images to test migration instead of
nested virt (and not only migration)
-> we'd loose support for non-Linux platforms like FreeBSD which we
would not if we used nested
* Hosting the test suite itself
- the main point to discuss here is whether the test suite should be part
of the main libvirt repo following QEMU's lead by example or should they
live inside a separate repo (a new one or as part of
libvirt-jenkins-ci [4]
-> the question here for QEMU folks is:
*"What was the rationale for QEMU to decide to have avocado-qemu as
part of the main repo?"*
* What framework to use for the test suite
- libvirt-tck because it already contains a bunch of very useful tests as
mentioned in the beginning
- using the avocado-vt plugin because that's what's the existing
libvirt-test-provider [1] is about
- pure avocado for its community popularity and continuous development and
once again follow QEMU leading by example
-> and again a question for QEMU folks:
*"What was QEMU's take on this and why did they decide to go with
avocado-qemu?"*
* Integrating the test suite with the main libvirt.git repo
- if we host the suite as part of libvirt-jenkins-ci as mentioned in the
previous section then we could make libvirt-jenkins-ci a submodule of
libvirt.git and enhance the toolchain by having something like 'make
integration' that would prepare the selected guests and execute the test
suite in them (only on demand)
Regards,
Erik
[1] https://github.com/autotest/tp-libvirt
[2] https://libvirt.org/testtck.html
[3] https://wiki.centos.org/QaWiki/CI/GettingStarted#head-a46ee49e8818ef9b502...
[4] https://github.com/libvirt/libvirt-jenkins-ci
5 years, 7 months
[libvirt] [glib PATCH v2 00/15] po: improve translation handling
by Daniel P. Berrangé
This applies the same improvements previously done in libvirt:
https://www.redhat.com/archives/libvir-list/2018-April/msg01004.html
https://www.berrange.com/posts/2018/11/29/improved-translation-po-file-ha...
The key problems with our current approach are:
- The pot & po files stored in GIT contain huge set of
annotations about source file names & line numbers.
These are out of date as soon as a change is commited
to git following a translation refresh. This makes
diffs impossible to meaningfully review, as they are
98% noise, 2% signal.
- The po file messages are sorted by source location,
so when we move code between files, or rename files,
the po file message order changes for no good reason.
This makes diffs even more impossible to review.
- The po files contain entries for all messages even
if most have no translation, bloating size of po/
data stored in git
- Whenever 'make dist' is run, it alters all the pot
and po files, so developers need to then reset their
content to match git HEAD manually. This is caused
by having auto-generated content (source file locations)
mixed in with the static content (the actual translated
strings)
After this series, we only minimized po files in git, with
the redundated & outdated source locations info stripped.
This stripped info is re-added automatically during build
to create the real .po files, that we distribute, and/or
upload to translators in Zanata.
As a result the po directory is smaller in size, and
when refreshing from Zanata, we have git commits that
clearly show *only* the altered translations, nothing
else. The importance of this cannot be overstated - by
having these clear diffs when doing this change in libvirt,
I discovered a serious bug in the Zanata client that has
been screwing up translations in every project that uses
Zanata by adding bogus "fuzzy" annotations.
The particularly attractive statistic:
194 files changed, 2607 insertions(+), 14146 deletions(-)
Changed in v2:
- Fix a number of problems with VPATH builds
Daniel P. Berrangé (15):
po: provide custom make rules for po file management
po: remove language list from zanata configuration
po: add rules for integration with zanata
po: minimize & canonicalize translations stored in git
po: minimize af am anp ar as ast bal be bg bn_IN
po: minimize bn bo br brx bs ca cs cy da de_CH
po: minimize de el en_GB eo es et eu fa fi
po: minimize fr gl gu he hi hr hu ia
po: minimize id ilo is it ja ka kk km kn ko
po: minimize kw_GB kw@kkcor kw kw@uccor ky lt lv mai mk ml
po: minimize mn mr ms nb nds ne nl nn nso
po: minimize or pa pl pt_BR pt ro ru si sk sl
po: minimize sq sr@latin sr sv ta te tg th tr tw
po: minimize uk ur vi wba yo zh_CN zh_HK zh_TW zu
po: refresh translations from zanata
.gitignore | 11 +--
autogen.sh | 3 +-
build-aux/minimize-po.pl | 37 ++++++++
configure.ac | 8 +-
libvirt-gconfig/Makefile.am | 1 +
libvirt-glib.spec.in | 2 +-
libvirt-glib/Makefile.am | 1 +
libvirt-gobject/Makefile.am | 1 +
m4/virt-gettext.m4 | 5 --
m4/virt-nls.m4 | 45 ++++++++++
mingw-libvirt-glib.spec.in | 2 +-
po/Makefile.am | 111 +++++++++++++++++++++++
po/{POTFILES.in => POTFILES} | 0
po/README.md | 76 ++++++++++++++++
po/af.mini.po | 20 +++++
po/af.po | 153 --------------------------------
po/am.mini.po | 20 +++++
po/am.po | 153 --------------------------------
po/anp.mini.po | 19 ++++
po/anp.po | 152 -------------------------------
po/ar.mini.po | 21 +++++
po/ar.po | 154 --------------------------------
po/as.mini.po | 20 +++++
po/as.po | 153 --------------------------------
po/ast.mini.po | 20 +++++
po/ast.po | 153 --------------------------------
po/bal.mini.po | 20 +++++
po/bal.po | 153 --------------------------------
po/be.mini.po | 21 +++++
po/be.po | 154 --------------------------------
po/bg.mini.po | 20 +++++
po/bg.po | 153 --------------------------------
po/bn.mini.po | 20 +++++
po/bn.po | 153 --------------------------------
po/bn_IN.mini.po | 20 +++++
po/bn_IN.po | 153 --------------------------------
po/bo.mini.po | 20 +++++
po/bo.po | 153 --------------------------------
po/br.mini.po | 20 +++++
po/br.po | 153 --------------------------------
po/brx.mini.po | 20 +++++
po/brx.po | 153 --------------------------------
po/bs.mini.po | 21 +++++
po/bs.po | 154 --------------------------------
po/{ca.po => ca.mini.po} | 152 ++++++++++++-------------------
po/cs.mini.po | 119 +++++++++++++++++++++++++
po/cs.po | 155 --------------------------------
po/cy.mini.po | 21 +++++
po/cy.po | 154 --------------------------------
po/da.mini.po | 20 +++++
po/da.po | 153 --------------------------------
po/de.mini.po | 20 +++++
po/de.po | 153 --------------------------------
po/de_CH.mini.po | 20 +++++
po/de_CH.po | 153 --------------------------------
po/el.mini.po | 20 +++++
po/el.po | 153 --------------------------------
po/{en_GB.po => en_GB.mini.po} | 141 ++++++++++-------------------
po/eo.mini.po | 20 +++++
po/eo.po | 153 --------------------------------
po/{es.po => es.mini.po} | 153 ++++++++++++--------------------
po/et.mini.po | 20 +++++
po/et.po | 153 --------------------------------
po/eu.mini.po | 20 +++++
po/eu.po | 153 --------------------------------
po/fa.mini.po | 20 +++++
po/fa.po | 153 --------------------------------
po/fi.mini.po | 44 +++++++++
po/fi.po | 154 --------------------------------
po/{fr.po => fr.mini.po} | 157 +++++++++++++--------------------
po/gl.mini.po | 20 +++++
po/gl.po | 153 --------------------------------
po/gu.mini.po | 20 +++++
po/gu.po | 153 --------------------------------
po/he.mini.po | 20 +++++
po/he.po | 153 --------------------------------
po/{hi.po => hi.mini.po} | 141 ++++++++++-------------------
po/hr.mini.po | 21 +++++
po/hr.po | 154 --------------------------------
po/hu.mini.po | 20 +++++
po/hu.po | 153 --------------------------------
po/ia.mini.po | 20 +++++
po/ia.po | 153 --------------------------------
po/id.mini.po | 20 +++++
po/id.po | 153 --------------------------------
po/ilo.mini.po | 20 +++++
po/ilo.po | 153 --------------------------------
po/is.mini.po | 20 +++++
po/is.po | 153 --------------------------------
po/it.mini.po | 20 +++++
po/it.po | 153 --------------------------------
po/{ja.po => ja.mini.po} | 141 ++++++++++-------------------
po/ka.mini.po | 20 +++++
po/ka.po | 153 --------------------------------
po/kk.mini.po | 20 +++++
po/kk.po | 153 --------------------------------
po/km.mini.po | 20 +++++
po/km.po | 153 --------------------------------
po/kn.mini.po | 20 +++++
po/kn.po | 153 --------------------------------
po/ko.mini.po | 20 +++++
po/ko.po | 153 --------------------------------
po/kw.mini.po | 20 +++++
po/kw.po | 153 --------------------------------
po/kw(a)kkcor.mini.po | 20 +++++
po/kw(a)kkcor.po | 153 --------------------------------
po/kw(a)uccor.mini.po | 20 +++++
po/kw(a)uccor.po | 153 --------------------------------
po/kw_GB.mini.po | 20 +++++
po/kw_GB.po | 153 --------------------------------
po/ky.mini.po | 20 +++++
po/ky.po | 153 --------------------------------
po/lt.mini.po | 21 +++++
po/lt.po | 154 --------------------------------
po/lv.mini.po | 21 +++++
po/lv.po | 154 --------------------------------
po/mai.mini.po | 20 +++++
po/mai.po | 153 --------------------------------
po/mk.mini.po | 20 +++++
po/mk.po | 153 --------------------------------
po/ml.mini.po | 20 +++++
po/ml.po | 153 --------------------------------
po/mn.mini.po | 20 +++++
po/mn.po | 153 --------------------------------
po/mr.mini.po | 20 +++++
po/mr.po | 153 --------------------------------
po/ms.mini.po | 20 +++++
po/ms.po | 153 --------------------------------
po/nb.mini.po | 20 +++++
po/nb.po | 153 --------------------------------
po/nds.mini.po | 20 +++++
po/nds.po | 153 --------------------------------
po/ne.mini.po | 20 +++++
po/ne.po | 153 --------------------------------
po/nl.mini.po | 20 +++++
po/nl.po | 153 --------------------------------
po/nn.mini.po | 20 +++++
po/nn.po | 153 --------------------------------
po/nso.mini.po | 20 +++++
po/nso.po | 153 --------------------------------
po/or.mini.po | 20 +++++
po/or.po | 153 --------------------------------
po/pa.mini.po | 20 +++++
po/pa.po | 153 --------------------------------
po/{pl.po => pl.mini.po} | 152 ++++++++++++-------------------
po/pt.mini.po | 20 +++++
po/pt.po | 153 --------------------------------
po/{pt_BR.po => pt_BR.mini.po} | 141 ++++++++++-------------------
po/ro.mini.po | 21 +++++
po/ro.po | 154 --------------------------------
po/ru.mini.po | 26 ++++++
po/ru.po | 155 --------------------------------
po/si.mini.po | 20 +++++
po/si.po | 153 --------------------------------
po/sk.mini.po | 20 +++++
po/sk.po | 153 --------------------------------
po/sl.mini.po | 21 +++++
po/sl.po | 154 --------------------------------
po/sq.mini.po | 20 +++++
po/sq.po | 153 --------------------------------
po/sr.mini.po | 21 +++++
po/sr.po | 154 --------------------------------
po/sr(a)latin.mini.po | 21 +++++
po/sr(a)latin.po | 154 --------------------------------
po/sv.mini.po | 20 +++++
po/sv.po | 153 --------------------------------
po/ta.mini.po | 20 +++++
po/ta.po | 153 --------------------------------
po/te.mini.po | 20 +++++
po/te.po | 153 --------------------------------
po/tg.mini.po | 20 +++++
po/tg.po | 153 --------------------------------
po/th.mini.po | 20 +++++
po/th.po | 153 --------------------------------
po/tr.mini.po | 20 +++++
po/tr.po | 153 --------------------------------
po/tw.mini.po | 19 ++++
po/tw.po | 152 -------------------------------
po/{uk.po => uk.mini.po} | 152 ++++++++++++-------------------
po/ur.mini.po | 20 +++++
po/ur.po | 153 --------------------------------
po/vi.mini.po | 20 +++++
po/vi.po | 153 --------------------------------
po/wba.mini.po | 19 ++++
po/wba.po | 152 -------------------------------
po/yo.mini.po | 19 ++++
po/yo.po | 152 -------------------------------
po/zanata.xml | 99 ---------------------
po/zh_CN.mini.po | 20 +++++
po/zh_CN.po | 153 --------------------------------
po/zh_HK.mini.po | 20 +++++
po/zh_HK.po | 153 --------------------------------
po/zh_TW.mini.po | 20 +++++
po/zh_TW.po | 153 --------------------------------
po/zu.mini.po | 20 +++++
po/zu.po | 153 --------------------------------
196 files changed, 2609 insertions(+), 14149 deletions(-)
create mode 100755 build-aux/minimize-po.pl
delete mode 100644 m4/virt-gettext.m4
create mode 100644 m4/virt-nls.m4
create mode 100644 po/Makefile.am
rename po/{POTFILES.in => POTFILES} (100%)
create mode 100644 po/README.md
create mode 100644 po/af.mini.po
delete mode 100644 po/af.po
create mode 100644 po/am.mini.po
delete mode 100644 po/am.po
create mode 100644 po/anp.mini.po
delete mode 100644 po/anp.po
create mode 100644 po/ar.mini.po
delete mode 100644 po/ar.po
create mode 100644 po/as.mini.po
delete mode 100644 po/as.po
create mode 100644 po/ast.mini.po
delete mode 100644 po/ast.po
create mode 100644 po/bal.mini.po
delete mode 100644 po/bal.po
create mode 100644 po/be.mini.po
delete mode 100644 po/be.po
create mode 100644 po/bg.mini.po
delete mode 100644 po/bg.po
create mode 100644 po/bn.mini.po
delete mode 100644 po/bn.po
create mode 100644 po/bn_IN.mini.po
delete mode 100644 po/bn_IN.po
create mode 100644 po/bo.mini.po
delete mode 100644 po/bo.po
create mode 100644 po/br.mini.po
delete mode 100644 po/br.po
create mode 100644 po/brx.mini.po
delete mode 100644 po/brx.po
create mode 100644 po/bs.mini.po
delete mode 100644 po/bs.po
rename po/{ca.po => ca.mini.po} (62%)
create mode 100644 po/cs.mini.po
delete mode 100644 po/cs.po
create mode 100644 po/cy.mini.po
delete mode 100644 po/cy.po
create mode 100644 po/da.mini.po
delete mode 100644 po/da.po
create mode 100644 po/de.mini.po
delete mode 100644 po/de.po
create mode 100644 po/de_CH.mini.po
delete mode 100644 po/de_CH.po
create mode 100644 po/el.mini.po
delete mode 100644 po/el.po
rename po/{en_GB.po => en_GB.mini.po} (51%)
create mode 100644 po/eo.mini.po
delete mode 100644 po/eo.po
rename po/{es.po => es.mini.po} (59%)
create mode 100644 po/et.mini.po
delete mode 100644 po/et.po
create mode 100644 po/eu.mini.po
delete mode 100644 po/eu.po
create mode 100644 po/fa.mini.po
delete mode 100644 po/fa.po
create mode 100644 po/fi.mini.po
delete mode 100644 po/fi.po
rename po/{fr.po => fr.mini.po} (59%)
create mode 100644 po/gl.mini.po
delete mode 100644 po/gl.po
create mode 100644 po/gu.mini.po
delete mode 100644 po/gu.po
create mode 100644 po/he.mini.po
delete mode 100644 po/he.po
rename po/{hi.po => hi.mini.po} (62%)
create mode 100644 po/hr.mini.po
delete mode 100644 po/hr.po
create mode 100644 po/hu.mini.po
delete mode 100644 po/hu.po
create mode 100644 po/ia.mini.po
delete mode 100644 po/ia.po
create mode 100644 po/id.mini.po
delete mode 100644 po/id.po
create mode 100644 po/ilo.mini.po
delete mode 100644 po/ilo.po
create mode 100644 po/is.mini.po
delete mode 100644 po/is.po
create mode 100644 po/it.mini.po
delete mode 100644 po/it.po
rename po/{ja.po => ja.mini.po} (55%)
create mode 100644 po/ka.mini.po
delete mode 100644 po/ka.po
create mode 100644 po/kk.mini.po
delete mode 100644 po/kk.po
create mode 100644 po/km.mini.po
delete mode 100644 po/km.po
create mode 100644 po/kn.mini.po
delete mode 100644 po/kn.po
create mode 100644 po/ko.mini.po
delete mode 100644 po/ko.po
create mode 100644 po/kw.mini.po
delete mode 100644 po/kw.po
create mode 100644 po/kw(a)kkcor.mini.po
delete mode 100644 po/kw(a)kkcor.po
create mode 100644 po/kw(a)uccor.mini.po
delete mode 100644 po/kw(a)uccor.po
create mode 100644 po/kw_GB.mini.po
delete mode 100644 po/kw_GB.po
create mode 100644 po/ky.mini.po
delete mode 100644 po/ky.po
create mode 100644 po/lt.mini.po
delete mode 100644 po/lt.po
create mode 100644 po/lv.mini.po
delete mode 100644 po/lv.po
create mode 100644 po/mai.mini.po
delete mode 100644 po/mai.po
create mode 100644 po/mk.mini.po
delete mode 100644 po/mk.po
create mode 100644 po/ml.mini.po
delete mode 100644 po/ml.po
create mode 100644 po/mn.mini.po
delete mode 100644 po/mn.po
create mode 100644 po/mr.mini.po
delete mode 100644 po/mr.po
create mode 100644 po/ms.mini.po
delete mode 100644 po/ms.po
create mode 100644 po/nb.mini.po
delete mode 100644 po/nb.po
create mode 100644 po/nds.mini.po
delete mode 100644 po/nds.po
create mode 100644 po/ne.mini.po
delete mode 100644 po/ne.po
create mode 100644 po/nl.mini.po
delete mode 100644 po/nl.po
create mode 100644 po/nn.mini.po
delete mode 100644 po/nn.po
create mode 100644 po/nso.mini.po
delete mode 100644 po/nso.po
create mode 100644 po/or.mini.po
delete mode 100644 po/or.po
create mode 100644 po/pa.mini.po
delete mode 100644 po/pa.po
rename po/{pl.po => pl.mini.po} (62%)
create mode 100644 po/pt.mini.po
delete mode 100644 po/pt.po
rename po/{pt_BR.po => pt_BR.mini.po} (54%)
create mode 100644 po/ro.mini.po
delete mode 100644 po/ro.po
create mode 100644 po/ru.mini.po
delete mode 100644 po/ru.po
create mode 100644 po/si.mini.po
delete mode 100644 po/si.po
create mode 100644 po/sk.mini.po
delete mode 100644 po/sk.po
create mode 100644 po/sl.mini.po
delete mode 100644 po/sl.po
create mode 100644 po/sq.mini.po
delete mode 100644 po/sq.po
create mode 100644 po/sr.mini.po
delete mode 100644 po/sr.po
create mode 100644 po/sr(a)latin.mini.po
delete mode 100644 po/sr(a)latin.po
create mode 100644 po/sv.mini.po
delete mode 100644 po/sv.po
create mode 100644 po/ta.mini.po
delete mode 100644 po/ta.po
create mode 100644 po/te.mini.po
delete mode 100644 po/te.po
create mode 100644 po/tg.mini.po
delete mode 100644 po/tg.po
create mode 100644 po/th.mini.po
delete mode 100644 po/th.po
create mode 100644 po/tr.mini.po
delete mode 100644 po/tr.po
create mode 100644 po/tw.mini.po
delete mode 100644 po/tw.po
rename po/{uk.po => uk.mini.po} (65%)
create mode 100644 po/ur.mini.po
delete mode 100644 po/ur.po
create mode 100644 po/vi.mini.po
delete mode 100644 po/vi.po
create mode 100644 po/wba.mini.po
delete mode 100644 po/wba.po
create mode 100644 po/yo.mini.po
delete mode 100644 po/yo.po
create mode 100644 po/zh_CN.mini.po
delete mode 100644 po/zh_CN.po
create mode 100644 po/zh_HK.mini.po
delete mode 100644 po/zh_HK.po
create mode 100644 po/zh_TW.mini.po
delete mode 100644 po/zh_TW.po
create mode 100644 po/zu.mini.po
delete mode 100644 po/zu.po
--
2.20.1
5 years, 7 months
