libvirt-devaddr: a new library for device address assignment
by Daniel P. Berrangé
We've been doing alot of refactoring of code in recent times, and also
have plans for significant infrastructure changes. We still need to
spend time delivering interesting features to users / applications.
This mail is to introduce an idea for a solution to an specific
area applications have had long term pain with libvirt's current
"mechanism, not policy" approach - device addressing. This is a way
for us to show brand new ideas & approaches for what the libvirt
project can deliver in terms of management APIs.
To set expectations straight: I have written no code for this yet,
merely identified the gap & conceptual solution.
The device addressing problem
=============================
One of the key jobs libvirt does when processing a new domain XML
configuration is to assign addresses to all devices that are present.
This involves adding various device controllers (PCI bridges, PCI root
ports, IDE/SCSI buses, USB controllers, etc) if they are not already
present, and then assigning PCI, USB, IDE, SCSI, etc, addresses to each
device so they are associated with controllers. When libvirt spawns a
QEMU guest, it will pass full address information to QEMU.
Libvirt, as a general rule, aims to avoid defining and implementing
policy around expansion of guest configuration / defaults, however, it
is inescapable in the case of device addressing due to the need to
guarantee a stable hardware ABI to make live migration and save/restore
to disk work. The policy that libvirt has implemented for device
addressing is, as much as possible, the same as the addressing scheme
QEMU would apply itself.
While libvirt succeeds in its goal of providing a stable hardware API,
the addressing scheme used is not well suited to all deployment
scenarios of QEMU. This is an inevitable result of having a specific
assignment policy implemented in libvirt which has to trade off mutually
incompatible use cases/goals.
When the libvirt addressing policy is not been sufficient, management
applications are forced to take on address assignment themselves,
which is a massive non-trivial job with many subtle problems to
consider.
Places where libvirt's addressing is insufficient for PCI include
* Setting up multiple guest NUMA nodes and associating devices to
specific nodes
* Pre-emptive creation of extra PCIe root ports, to allow for later
device hotplug on PCIe topologies
* Determining whether to place a device on a PCI or PCIe bridge
* Controlling whether a device is placed into a hotpluggable slot
* Controlling whether a PCIe root port supports hotplug or not
* Determining whether to places all devices on distinct slots or
buses, vs grouping them all into functions on the same slot
* Ability to expand the device addressing without being on the
hypervisor host
Libvirt wishes to avoid implementing many different address assignment
policies. It also wishes to keep the domain XML as a representation
of the virtual hardware, not add a bunch of properties to it which
merely serve as tunable input parameters for device addressing
algorithms.
There is thus a dilemma here. Management applications increasingly
need fine grained control over device addressing, while libvirt
doesn't want to expose fine grained policy controls via the XML.
The new libvirt-devaddr API
===========================
The way out of this is to define a brand new virt management API
which tackles this specific problem in a way that addresses all the
problems mgmt apps have with device addressing and explicitly
provides a variety of policy impls with tunable behaviour.
By "new API", I actually mean an entirely new library, completely
distinct from libvirt.so, or anything else we've delivered so
far. The closest we've come to delivering something at this kind
of conceptual level, would be the abortive attempt we made with
"libvirt-builder" to deliver a policy-driven API instead of mechanism
based. This proposal is still quite different from that attempt.
At a high level
* The new API is "libvirt-devaddr" - short for "libvirt device addressing"
* As input it will take
1. The guest CPU architecture and machine type
2. A list of global tunables specifying desired behaviour of the
address assignment policy
3. A minimal list of devices needed in the virtual machine, with
optional addresses and optional per-device tunables to override
the global tunables
* As output it will emit
1. fully expanded list of devices needed in the virtual machine,
with addressing information sufficient to ensure stable hardware ABI
Initially the API would implement something that behaves the same
way as libvirt's current address assignment API.
The intended usage would be
* Mgmt application makes a minimal list of devices they want in
their guest
* List of devices is fed into libvirt-devaddr API
* Mgmt application gets back a full list of devices & addresses
* Mgmt application writes a libvirt XML doc using this full list &
addresses
* Mgmt application creates the guest in libvirt
IOW, this new "libvirt-devaddr" API is intended to be used prior to
creating the XML that is used by libvirt. The API could also be used
prior to needing to hotplug a new device to an existing guest.
This API is intended to be a deliverable of the libvirt project, but
it would be completely independent of the current libvirt API. Most
especially note that it would NOT use the domain XML in any way.
This gives applications maximum flexibility in how they consume this
functionality, not trying to force a way to build domain XML.
It would have greater freedom in its API design, making different
choices from libvirt.so on topics such as programming language (C vs
Go vs Python etc), API stability timeframe (forever stable vs sometimes
changing API), data formats (structs, vs YAML/JSON vs XML etc), and of
course the conceptual approach (policy vs mechanism)
The expectation is that this new API would be most likely to be
consumed by KubeVirt, OpenStack, Kata, as the list of problems shown
earlier is directly based on issues seen working with KubeVirt &
OpenStack in particular. It is not limited to these applications and
is broadly useful as conceptual thing.
It would be a goal that this API should also be used by libvirt
itself to replace its current internal device addressing impl.
Essentially the new API should be seen as a way to expose/extract
the current libvirt internal algorithm, making it available to
applications in a flexible manner. I don't anticipate actually copying
the current addressing code in libvirt as-is, but it would certainly
serve as reference for the kind of logic we need to implement, so you
might consider it a "port" or "rewrite" in some very rough sense.
I think this new API concept is a good way for the project make a start
in using Go for libvirt. The functionality covered has a clearly defined
scope limit, making it practical to deliver a real impl in a reasonably
short time frame. Extracting this will provide a real world benefit to
our application consumers, solving many long standing problems they have
with libvirt, and thus justify the effort in doing this work in libvirt
in a non-C language. The main question mark would be about how we might
make this functionality available to Python apps if we chose Go. It is
possible to expose a C API from Go, and we would need this to consume it
from libvirt. There is then the need to manually write a Python API binding
which is tedious work.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
4 years, 6 months
- 7
- 14
[libvirt-jenkins-ci PATCH] guests: allow for container image inheritance
by Daniel P. Berrangé
Currently when creating a Dockerfile for a container, we include the
full set of base packages, along with the packages for the project
itself. If building a Perl binding, this would require us to install
the base package, libvirt packages and Perl packages. With the use
of the "--inherit libvirt-fedora-30" arg, it is possible to have a
dockerfile that only adds the Perl packages, getting everything
else from the parent image.
For example, a full Dockerfile for libvirt-go would thus be:
FROM libvirt-centos-7:latest
RUN yum install -y \
golang && \
yum autoremove -y && \
yum clean all -y
Note there is no need to set ENV either, as that's inherited from the
parent container.
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
guests/lcitool | 108 ++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 98 insertions(+), 10 deletions(-)
diff --git a/guests/lcitool b/guests/lcitool
index 689a8cf..b3afb6a 100755
--- a/guests/lcitool
+++ b/guests/lcitool
@@ -396,6 +396,12 @@ class Application:
help="target architecture for cross compiler",
)
+ def add_inherit_arg(parser):
+ parser.add_argument(
+ "-i", "--inherit",
+ help="inherit from an intermediate container image",
+ )
+
def add_wait_arg(parser):
parser.add_argument(
"-w", "--wait",
@@ -442,6 +448,7 @@ class Application:
add_hosts_arg(dockerfileparser)
add_projects_arg(dockerfileparser)
add_cross_arch_arg(dockerfileparser)
+ add_inherit_arg(dockerfileparser)
def _execute_playbook(self, playbook, hosts, projects, git_revision):
base = Util.get_base()
@@ -644,11 +651,11 @@ class Application:
with open(keyfile, "r") as r:
return r.read().rstrip()
- def _dockerfile_build_varmap(self, facts, mappings, pip_mappings, projects, cross_arch):
+ def _dockerfile_build_varmap(self, facts, mappings, pip_mappings, projects, cross_arch, needbase):
if facts["package_format"] == "deb":
- varmap = self._dockerfile_build_varmap_deb(facts, mappings, pip_mappings, projects, cross_arch)
+ varmap = self._dockerfile_build_varmap_deb(facts, mappings, pip_mappings, projects, cross_arch, needbase)
if facts["package_format"] == "rpm":
- varmap = self._dockerfile_build_varmap_rpm(facts, mappings, pip_mappings, projects, cross_arch)
+ varmap = self._dockerfile_build_varmap_rpm(facts, mappings, pip_mappings, projects, cross_arch, needbase)
varmap["package_manager"] = facts["package_manager"]
varmap["cc"] = facts["cc"]
@@ -662,7 +669,7 @@ class Application:
return varmap
- def _dockerfile_build_varmap_deb(self, facts, mappings, pip_mappings, projects, cross_arch):
+ def _dockerfile_build_varmap_deb(self, facts, mappings, pip_mappings, projects, cross_arch, needbase):
package_format = facts["package_format"]
package_manager = facts["package_manager"]
os_name = facts["os_name"]
@@ -682,7 +689,10 @@ class Application:
# We need to add the base project manually here: the standard
# machinery hides it because it's an implementation detail
- for project in projects + ["base"]:
+ pkgprojects = projects
+ if needbase:
+ pkgprojects = projects + ["base"]
+ for project in pkgprojects:
for package in self._projects.get_packages(project):
cross_policy = "native"
for key in cross_keys:
@@ -730,7 +740,7 @@ class Application:
return varmap
- def _dockerfile_build_varmap_rpm(self, facts, mappings, pip_mappings, projects, cross_arch):
+ def _dockerfile_build_varmap_rpm(self, facts, mappings, pip_mappings, projects, cross_arch, needbase):
package_format = facts["package_format"]
package_manager = facts["package_manager"]
os_name = facts["os_name"]
@@ -744,7 +754,10 @@ class Application:
# We need to add the base project manually here: the standard
# machinery hides it because it's an implementation detail
- for project in projects + ["base"]:
+ pkgprojects = projects
+ if needbase:
+ pkgprojects = projects + ["base"]
+ for project in pkgprojects:
for package in self._projects.get_packages(project):
for key in keys:
if key in mappings[package]:
@@ -791,7 +804,7 @@ class Application:
return varmap
- def _dockerfile_format(self, facts, cross_arch, varmap):
+ def _dockerfile_base_format(self, facts, cross_arch, varmap):
package_format = facts["package_format"]
package_manager = facts["package_manager"]
os_name = facts["os_name"]
@@ -931,6 +944,74 @@ class Application:
ENV CONFIGURE_OPTS "--host={cross_abi}"
""").format(**varmap))
+ def _dockerfile_child_format(self, facts, cross_arch, inherit, varmap):
+ package_format = facts["package_format"]
+ package_manager = facts["package_manager"]
+ os_name = facts["os_name"]
+ os_version = facts["os_version"]
+
+ print("FROM {}".format(inherit))
+
+ commands = []
+
+ if package_format == "deb":
+ commands.extend([
+ "export DEBIAN_FRONTEND=noninteractive",
+ "{package_manager} update",
+ "{package_manager} install --no-install-recommends -y {pkgs}",
+ "{package_manager} autoremove -y",
+ "{package_manager} autoclean -y",
+ ])
+ elif package_format == "rpm":
+ commands.extend([
+ "{package_manager} install -y {pkgs}",
+ ])
+
+ # openSUSE doesn't seem to have a convenient way to remove all
+ # unnecessary packages, but CentOS and Fedora do
+ if os_name == "OpenSUSE":
+ commands.extend([
+ "{package_manager} clean --all",
+ ])
+ else:
+ commands.extend([
+ "{package_manager} autoremove -y",
+ "{package_manager} clean all -y",
+ ])
+
+
+ script = "\nRUN " + (" && \\\n ".join(commands)) + "\n"
+ sys.stdout.write(script.format(**varmap))
+
+ if cross_arch:
+ cross_commands = []
+
+ # Intentionally a separate RUN command from the above
+ # so that the common packages of all cross-built images
+ # share a Docker image layer.
+ if package_format == "deb":
+ cross_commands.extend([
+ "export DEBIAN_FRONTEND=noninteractive",
+ "{package_manager} update",
+ "{package_manager} install --no-install-recommends -y {cross_pkgs}",
+ "{package_manager} autoremove -y",
+ "{package_manager} autoclean -y",
+ ])
+ elif package_format == "rpm":
+ cross_commands.extend([
+ "{package_manager} install -y {cross_pkgs}",
+ "{package_manager} clean all -y",
+ ])
+
+ cross_script = "\nRUN " + (" && \\\n ".join(cross_commands)) + "\n"
+ sys.stdout.write(cross_script.format(**varmap))
+
+ if "pip_pkgs" in varmap:
+ sys.stdout.write(textwrap.dedent("""
+ RUN pip3 install {pip_pkgs}
+ """).format(**varmap))
+
+
def _action_dockerfile(self, args):
mappings = self._projects.get_mappings()
pip_mappings = self._projects.get_pip_mappings()
@@ -947,6 +1028,7 @@ class Application:
os_version = facts["os_version"]
os_full = os_name + os_version
cross_arch = args.cross_arch
+ inherit = args.inherit
if package_format not in ["deb", "rpm"]:
raise Exception("Host {} doesn't support Dockerfiles".format(host))
@@ -983,8 +1065,14 @@ class Application:
)
)
- varmap = self._dockerfile_build_varmap(facts, mappings, pip_mappings, projects, cross_arch)
- self._dockerfile_format(facts, cross_arch, varmap)
+ needbase = True
+ if inherit is not None:
+ needbase = False
+ varmap = self._dockerfile_build_varmap(facts, mappings, pip_mappings, projects, cross_arch, needbase)
+ if inherit is None:
+ self._dockerfile_base_format(facts, cross_arch, varmap)
+ else:
+ self._dockerfile_child_format(facts, cross_arch, inherit, varmap)
def run(self):
args = self._parser.parse_args()
--
2.24.1
4 years, 6 months
- 2
- 5
[libvirt PATCH 0/3] qemu: block: basic implementation of deletion of external snapshots
by Pavel Mores
Deleting external snapshots has been unimplemented so far. This series aims
to enable limited functionality in that direction, handling just the common
cases for now. The intention is to subsequently build upon this to eventually
cover the more complex/exotic cases as well.
Please refer to the commit message of the final commit for a detailed
description of features and limitations of this change.
Pavel Mores (3):
qemu: block: factor implementation out of qemuDomainBlockCommit()
qemu: block: factor implementation out of qemuDomainBlockJobAbort()
qemu: block: external snapshot-delete implementation for
straightforward cases
src/qemu/qemu_driver.c | 295 +++++++++++++++++++++++++++++++++--------
1 file changed, 241 insertions(+), 54 deletions(-)
--
2.24.1
4 years, 7 months
- 2
- 11
[RFC 00/29] RFC: Generate object-model code based on relax-ng files
by Shi Lei
Outline
=========
In libvirt world, objects(like domain, network, etc.) are described with two
representations: structures in c-language and XML specified by relax-ng.
Since c-language-implementation and xml restricted by relax-ng are merely
the different expressions of the same object-model, we can make a tool to
generate those C-language codes based on relax-ng files.
RNG2C tool
===========
RNG2C is a python tool which can translate hierarchy of notations
in relax-ng into c-language structures. In general, <element> is converted to
struct in C, struct's members derive from its <attribute>s and its sub <element>s;
<choice> with multiple <value>s is converted to enum in C; <data> are converted
into various builtin-type in C, such as int, char *.
Also, RNG2C can generate relative conversion functions: vir[XXX]ParseXML,
vir[XXX]FormatBuf and vir[XXX]Clear.
These structures and functions can be used to replace hardcoded codes in current
libvirt source.
The tool RNG2C has three subcommands: 'list', 'show' and 'generate'. The 'list' and
'show' are used for previewing generated code, and the 'generate' is prepared to be
invoked by Makefile.
1) list - list all types generated by this tool.
Example:
# ./rng2c/go list
SHORT_ID META LOCATION
-------- ---- ----------
3df63b7a String String
9b81a5f5 UInt UInt
9d10c738 Enum /basictypes.rng/virOnOff.define/choice
5a024c44 Struct /network.rng/network.define/network.element/dns.element
f6f0c927 Struct /network.rng/network.define/network.element
... ...
2) show - show details of the target type. The target is specified with
SHORT_ID or LOCATION. Option '-k' specifies kinds of output information,
'scpf' stands for [s]tructure | [c]learfunc | [p]arsefunc | [f]ormatfunc.
Example:
# ./rng2c/go show 5a024c44 -kscpf
Or
# ./rng2c/go show /network.rng/network.define/network.element/dns.element -kscpf
It will show four aspects of this Type: the directive of it; the declaration
and implementation of its structure; code of its clearfunc; code of its parsefunc;
code of its formatfunc.
3) generate - just like show, but it dosen't print information on screen. It is
only for Makefile. It will generate c-language-code, such as struct/enum/
parsefunc/etc., to take the place of the relative hardcoded stuff.
Problems
=========
If we replace hardcoded code with generated code directly, there're still
several problems.
Problem1 - name convension
Generated code follows specific name convension, now there're some differences
between generated code and current libvirt codes.
For example, generated struct's name will be like virXXXDef, parsefunc's name
will be like virXXXDefParseXML; the 'XXX' is a kind of Camel-Case name which
implies the struct's place in the hierarchy.
But in current code, there're many variations of name.
So if we replace current codes with generated codes directly, it requires
a lot of modifications in the caller codes.
Problem2 - error-checking codes in the parsefunc
Most of current parse functions have error-checking codes after parsing XML
code. These code lines validate existence and parsing results.
Now RNG2C can't generate error-checking codes in the parsefunc. So we
should find a way to handle it.
Problem3 - hierarchy consistence
The RNG2C generate a 'struct'(C code) for each 'element'(relax-ng). Most
of current code follows this convention, but there're still some exceptions.
For example, the element 'bridge'(in network.rng) should be translated into a
struct called 'virNetworkBridgeDef'; but in fact, no this struct in the current
codes, and its members (stp/delay/...) are exposed to the struct 'virNetworkDef'.
Problem4 - target output files for generated code
The current codes to be replaced exist in various source files. They
include other source files, and are included by other source files. So we
should specify proper target files for generated codes to take the place of
old codes and inherit those original relationships.
Directive
===========
To solve these problems, the directive mechanism is introduced.
stage1 stage2
relax-ng ---------> directives(json) ---------> c-language-code
^ |
| |
|---------------------|
feedback-injection
The basic process of generating codes includes two internal stages:
Stage1: RNG2C parses the content of relax-ng files and generates directives.
Stage2: RNG2C generates c-language code based on directives. So the code's form
depends on directives directly.
Injecting directives with new values into relax-ng files can override the
default directives of stage1. Then in the stage2, the final c-code will be different.
Directives are in the form of json and then wrapped as an special XML comment.
So when common relax-ng parsers parse these relax-ng files, these directives will be
ignored. Only RNG2C can recognize and handle them.
For example:
We want to replace the hardcoded virNetworkDNSHostDef with the generated structure.
The original hardcoded structure is as below:
typedef struct _virNetworkDNSHostDef virNetworkDNSHostDef;
typedef virNetworkDNSHostDef *virNetworkDNSHostDefPtr;
struct _virNetworkDNSHostDef {
virSocketAddr ip;
size_t nnames;
char **names;
};
Take several steps as below:
1) Execute subcommand list to find it
# ./rng2c/go list | grep dns | grep host
SHORT_ID META LOCATION
7892979d Struct /network.rng/network.define/network.element/dns.element/host.element
2) Execute subcommand show with SHORT-ID to check its generated structure
# ./rng2c/go show 7892979d -ks
###### Directive ######
<!-- VIRT:DIRECTIVE {
"id": "7892979d7e7a89882e9a324d95198d05d851d02b",
"location": "/network.rng/network.define/network.element/dns.element/host.element",
"name": "virNetworkDNSHostDef",
"meta": "Struct",
"members": [
{"id": "ip", "type": "String:b48b4186"},
{"id": "hostname", "more": true, "type": "String:7d3736b9"}
]
} -->
###### Structure (Disabled: NO OUTPUT for "structure".) ######
[.h]
typedef struct _virNetworkDNSHostDef virNetworkDNSHostDef;
typedef virNetworkDNSHostDef *virNetworkDNSHostDefPtr;
struct _virNetworkDNSHostDef {
virSocketAddr ip;
size_t nhostnames;
char **hostnames;
};
We can see, the name of the generated struct depends on root property "name"
of the directive. Fortunately, it's the same and we don't need override it.
There're two differences between generated structure and hardcoded stuff:
In original struct, names of the second and third member are 'nnames' and
'names'; but in generated code, they are 'nhostnames' and 'hostnames'.
These names are controlled by this part:
"members": [{"id": "hostname", ...}].
3) Override properties of the directive and reinject it into the relax-ng
Based on "location" of the directive, we find the injection point and only need
to fill changed parts, just as below:
<!-- VIRT:DIRECTIVE {
"members": [{"id": "hostname", "name": "name"}]
} -->
<element name="host">
... ...
We need to specify which member will be altered by property "id".
4) Re-execute subcommand 'show' to confirm the sameness
After we bridge the gap between the generated structure and the hardcoded
stuff, we can perform the substitution.
Directive properties
=====================
The schema of directive are defined in rng2c/schema.json.
Directive consists of properties. Properties retain or derive from the notations
in the relax-ng.
1) root properties
id - string, READ-ONLY.
Unique identity of the type.
The id is generated by SHA-1. Its leftmost 8 digits are taken out as SHORT-ID.
location - string, READ-ONLY.
The location where the type derive from.
When injecting a directive into relax-ng file, the injection point is based on
this location.
tag - string, READ-ONLY.
The type's source tag. It can be 'element', 'data' or 'choice'.
name - string.
The type's name.
For struct and enum, name is generated automatically. For struct, its default
form is vir[XXX]Def; for enum, its default form is vir[XXX]Type. The main
part 'XXX' represents its position in the hierarchy.
If name is specified, it will override the default name.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {"name": "virNetDevIPRoute"} -->
<element name="route">
... ...
Then the name of the generated struct will be 'virNetDevIPRoute'.
This root property 'name' can be used to solve Problem1.
meta - string.
The type's meta.
For struct, it is 'Struct'; for enum, it is 'Enum'; for builtins,
it is the builtin's meta.
Meta can be overrided. For example, inject directive as below:
<define name="UUID">
<!-- VIRT:DIRECTIVE {
"meta": "UChars",
"structure": {"array.size": "VIR_UUID_BUFLEN"}
} -->
<data type="string">
... ...
Then this builtin will change into UChars from String(i.e. char *).
Note: RNG2C don't generate implementation for builtin, they need to be
hardcoded in libvirt source.
unpack - boolean. Only for struct.
Only valid for struct. On setting it true, this type will expose its members
to its upper-level struct rather than act as a standalone struct.
For example, inject directive on element 'bridge' as below:
<!-- VIRT:DIRECTIVE {
"unpack": true,
"members": [ ... ... ]
} -->
<element name="bridge">
So struct virNetworkBridgeDef will not appear. All of its members, such as
bridgeName/bridgeZone/stp/delay, are exposed to its parent struct virNetworkDef.
The 'unpack' and 'pack' can be used to solve Problem3.
pack - boolean.
It can only work on <define> in the relax-ng.
Setting it true, children of <define> are converted into members; and then
these members are packed into a struct for reuse.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"name": "virPCIDeviceAddress",
"pack": true,
... ...
} -->
<define name="pciaddress">
... ...
The <define name="pciaddress"> will be packed as a struct virPCIDeviceAddress
which name is specified by the property "name".
union - string.
It can only work on <choice> in the relax-ng.
By default, all children of <choice> will act as members and be exposed to
the upper-level struct. When setting 'union' a non-null string which is called
union-id, those members will be packed into a struct; then, a new member
defined by this struct will be exposed to the upper-level struct.
The new member's id and name are specified by the union-id.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"union": "if",
"name": "virNetworkForwardIfDef"
} -->
<choice>
<group>
<zeroOrMore>
<!-- VIRT:DIRECTIVE {"unpack": true} -->
<element name='interface'>
...
</group>
<group>
<zeroOrMore>
<!-- VIRT:DIRECTIVE {"unpack": true} -->
<element name='address'>
...
</group>
</choice>
A new struct virNetworkForwardIfDef will be created; then a new member named 'if'
is exposed to the parent struct.
namespace - boolean. Only for struct.
By default, there's no stuff about namespace in the struct. When setting
it true, additional namespace members will be inserted into the struct.
And the struct's clearfunc, parsefunc and formatfunc will include relative codes
to handle these namespace stuff.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"namespace": true,
... ...
} -->
<element name="network">
Then in the struct virNetworkDef, there're two additional members:
typedef virNetworkDef *virNetworkDefPtr;
struct _virNetworkDef {
... ...
void *namespaceData;
virXMLNamespace ns;
}
In the virNetworkDefClear, there're the codes to invoke ns.free;
in the virNetworkDefParseXML, there're the codes to invoke ns.parse;
in the virNetworkDefFormatBuf, there're the codes to invoke ns.format.
values - array. Only for enum.
All values of an enum except 'none' and 'last'.
It can be overrided.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"name": "virNetworkForwardType",
"values": ["nat", "route", "open", "bridge",
"private", "vepa", "passthrough", "hostdev"]
} -->
<choice>
... ...
<value>passthrough</value>
<value>private</value>
<value>vepa</value>
<value>hostdev</value>
</choice>
Generally the property 'values' doesn't need to be overrided,
only if the order of values in the enum need to be adjusted.
members - array. Only for struct.
Property 'members' is used to display all the members of the struct.
But when altering a member, only fill in this member's 'id' and
properties to be changed.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "localPtr", "name": "localPTR"},
{"id": "tftp", "name": "tftproot"}
]
} -->
<element name="ip">
Specify the member "localPtr" and "tftp" by their id, then only
these two members will be altered. In this case, their names are changed.
For more member properties, refer to 6)
PRESERVE and APPLY - string.
By default, directives take effect on the injection point.
But sometimes, we need preserve a directive at first and then
apply it somewhere else.
APPLY corresponds to PRESERVE through their value called preserve-id.
This preserve-id can be any form of string.
These two properties can be used on two notations, <choice> and <ref>.
Here's an example for using them on <choice>, inject directive as below:
<define name="virtualPortProfile">
<!-- VIRT:DIRECTIVE {
"PRESERVE": "virtualport.element",
"name": "virNetDevVPortProfile",
... ...
} -->
<choice>
<group>
<!-- VIRT:DIRECTIVE {"APPLY": "virtualport.element"} -->
<element name="virtualport">
</group>
<group>
<!-- VIRT:DIRECTIVE {"APPLY": "virtualport.element"} -->
<element name="virtualport">
</group>
... ...
In this case, there're many <element name="virtualport"> notation
under <choice>. We can define a preserve directive and apply it to all
the <element name="virtualport">.
Here's an example for using them on <ref>, inject PRESERVE as below:
<!-- VIRT:DIRECTIVE {
"PRESERVE": "port.element",
"unpack": true,
... ...
"members": [{"id": "isolated", "name": "isolatedPort"}]
} -->
<ref name="portOptions"/>
... ...
And in <define>, inject APPLY as below:
<define name="portOptions">
<!-- VIRT:DIRECTIVE {"APPLY": "port.element"} -->
<element name="port">
... ...
It is useful in a case: a <define> is referenced by several <ref>s.
We can define different directives on different <ref>s and expect different
effects when they are applied in the <define>.
TOUCH - boolean.
Request to handle the target <define> explicitly.
By default, the process of RNG2C starts at <start>, and handles a <define>
only when a <ref> references to it. If a <define> isn't touched, RNG2C cannot
find it.
Inject a directive with TOUCH on a <define> to touch it.
For example, the <define> 'zpciaddress' hasn't been referenced in the process
of parsing network.rng (also networkcommon.rng and basictypes.rng), but we hope
that this notation will be handle.
Just inject the directive as below:
<!-- VIRT:DIRECTIVE {"TOUCH": true} -->
<define name="zpciaddress">
<element name="zpci">
... ...
Then it can be discovered when executing subcommand 'list'.
2) structure properties
output - string. Only for struct and enum.
Control whether to output the structure code and where to output those code.
Refer to 7)
enum.default - string. Only for enum.
By default, for enum, first item's name is 'none'.
Override it with other name.
For example, inject directive as below:
<define name="macTableManager">
<!-- VIRT:DIRECTIVE {
"name": "virNetworkBridgeMACTableManagerType",
"structure": {"enum.default": "default"}
} -->
<choice>
<value>kernel</value>
... ...
Then the declaration of enum virNetworkBridgeMACTableManagerType will be like:
typedef enum {
VIR_NETWORK_BRIDGE_MAC_TABLE_MANAGER_DEFAULT = 0,
VIR_NETWORK_BRIDGE_MAC_TABLE_MANAGER_KERNEL,
... ...
} virNetworkBridgeMACTableManagerType;
The first item is suffixed with '_DEFAULT' rather than '_NONE'.
array.size - string. Only for char array or unsigned char array.
It specifies length of array. The value can be a macro or a literal number.
For example, inject directive as below:
<define name="UUID">
<choice>
<!-- VIRT:DIRECTIVE {
"meta": "UChars",
"structure": {"array.size": "VIR_UUID_BUFLEN"}
} -->
<data type="string">
... ...
Then it will generate an array type:
unsigned char XXX[VIR_UUID_BUFLEN];
3) clearfunc properties
output - string. Only for struct.
Control whether to output the clearfunc code and where to output those code.
Refer to 7)
name - string.
The clearfunc's name.
By default, its name is based type's name and suffixed 'Clear'.
It can be overrided with any other name.
args - array.
Extra formal args for clearfunc. But now it hasn't been used.
4) parsefunc properties
The parsefunc's declaration and implementation are controlled by properties.
The basic form of parsefunc's declaration is as below:
int
${parsefunc.name}(xmlNodePtr curnode,
${type.name}Ptr def,
xmlXPathContextPtr ctxt, /* omit if args.noctxt is true */
const char *instanceName, /* exist if args.instname is true */
${parent.name}Ptr parent, /* exist if args.parent is true */
${args}); /* exist if args is NOT none */
output - string. Only for struct.
Control whether to output the parsefunc code and where to output those code.
Refer to 7)
name - string.
The parsefunc's name.
By default, its name is based type's name and suffixed 'ParseXML'.
It can be overrided with any other name.
args - array.
Extra formal args for parsefunc.
Each arg can have four properties: name, type, pointer and ctype.
The first three are according to the usage of namesake properties in member.
And ctype is an alternate to type when type can't express the argument's type.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"parsefunc": {
"args": [{"name": "partialOkay", "type": "Bool"}]
}
} -->
<element name="txt">
... ...
Then the args of parsefunc will be appended by a arg, as below:
int
virNetworkDNSTxtDefParseXML(xmlNodePtr curnode,
virNetworkDNSTxtDefPtr def,
xmlXPathContextPtr ctxt,
bool partialOkay);
args.noctxt - boolean.
By default, there's a arg 'ctxt' in the function arguments.
When 'args.noctxt' is set to true, omit ctxt from the arguments.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"parsefunc": {
"args.noctxt": true,
}
} -->
<element name="txt">
... ...
Then the arg 'ctxt' will be removed from the default args.
int
virNetworkDNSTxtDefParseXML(xmlNodePtr curnode,
virNetworkDNSTxtDefPtr def);
At the same time, the generated implementation of
virNetworkDNSTxtDefParseXML will be different according to
this property.
args.instname - boolean.
By default, there's no instance-name in the function arguments.
When it is set to true, add instname into the arguments.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"parsefunc": {
"args.instname": true,
}
} -->
<element name="txt">
... ...
Then the arg 'instname' will be added into the args.
int
virNetworkDNSTxtDefParseXML(xmlNodePtr curnode,
virNetworkDNSTxtDefPtr def,
xmlXPathContextPtr ctxt,
const char *instanceName);
args.parent - boolean.
By default, there's no parent struct pointer name in the function arguments.
When it is set to true, add parent struct pointer into the arguments.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"parsefunc": {
"args.parent": true
}
} -->
<element name="dhcp">
Then the arg 'parent' will be added into the args.
int
virNetworkIPDHCPDefParseXML(xmlNodePtr curnode,
virNetworkIPDHCPDefPtr def,
xmlXPathContextPtr ctxt,
virNetworkIPDefPtr parentdef);
post - boolean.
By default, there's no post hook in the parse function.
This property enables a post hook for parsefunc. The hook is invoked after
all members have been parsed. The post hook function's name is based on
parsefunc's name, and suffxied with 'Post'.
To solve the foregoing Problem2, extract error-checking code from the current
parsefunc and move them to the post hook function, then current parsefunc
can be replaced with generated parsefunc.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"parsefunc": {"post": true}
} -->
<element name="dns">
... ...
The declaration of the post hook will be generated, as below:
int
virNetworkDNSDefParseXMLPost(xmlNodePtr curnode,
virNetworkDNSDefPtr def,
xmlXPathContextPtr ctxt,
const char *enableStr,
const char *forwardPlainNamesStr,
int nForwarderNodes,
int nTxtNodes,
int nSrvNodes,
int nHostNodes);
Implement it. It will be invoked by generated parsefunc automatically.
post.notmpvars - boolean. It is valid only when post is set.
By default, all the temporary variables in the parsefunc will be passed to
the post hook function. When it is set to true, there's no those temporary
variables as arguments in the post hook functions.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"parsefunc": {"post": true, "post.notmpvars": true}
} -->
<element name="dns">
... ...
The declaration of the post hook will be generated, as below:
int
virNetworkDNSDefParseXMLPost(xmlNodePtr curnode,
virNetworkDNSDefPtr def,
xmlXPathContextPtr ctxt);
Now all tmpvars are removed from the args of parse hook function.
5) formatfunc properties
output - string. Only for struct.
Control whether to output the formatfunc code and where to output those code.
Refer to 7)
name - string.
The formatfunc's name.
By default, its name is based type's name and suffixed 'FormatBuf'.
It can be overrided with any other name.
args - array.
Extra formal args for formatfunc. Just like the args of parsefunc.
shorthand.ignore - boolean.
By default, formatfunc adopts shorthand closing tag notation when there're no
elements to output. If shorthand.ignore is set to true, this default
optimization will be prevented.
For example, virNetworkIPDefFormat does output <ip ...></ip> rather than
<ip .../> when there're no child element 'tftp' and 'dhcp'.
So this property is used to meet the need.
order - array.
It is a array which item is member's id.
In formatfunc, the relative order of formatting members is sensitive.
If the order is not correct, setting this property to re-arrange the order.
And when the array includes only part of ids, other members will follow and
keep their original relative order.
For example, virPortGroupDefFormat has a special order to format its members.
<!-- VIRT:DIRECTIVE {
"formatfunc": {
"order": ["name", "default", "trustGuestRxFilters", "vlan"]
}
} -->
Then format order will be altered according to this.
6) member properties
Member properties can only appear in the members' array. Use these properties
to override the struct members' default settings.
id - string, READ-ONLY.
When id is set, this property specify the member to be changed.
When id is ignored, it means this member is a new member and needed to append
to the struct's members.
For example, inject a directive as below:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "dev", "name": "devname"},
{"name": "connections", "type": "Int"}
]
} -->
<element name='pf'>
... ...
The the member 'dev' will be altered; and then, a new member 'connections' will
be added. For a new member, its type must be specified.
name - string.
Member's name.
By default, name is the same with id.
It can be overrided with any name.
opt - boolean.
Whether optional or not. In the parsefunc, non-existence for optional member
is legal; otherwise, there's a error-checking to check its existence and
report error.
For example, inject a directive as below:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "value", "opt": true}
]
} -->
<element name="txt">
... ...
Then in the virNetworkDNSTxtDefParseXML, it's ok when the notation 'value'
is missing.
more - boolean.
When it is set, the member is a list.
For example, inject a directive as below:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "pf", "more": true}
]
} -->
<element name="forward">
... ...
Then the member 'pf' will be declared as below:
size_t npfs;
virNetworkForwardPfDefPtr pfs;
tag - string.
Display the source's tag.
When a new member is to add, it can be specified.
type - string.
Member's type. Its form is "meta[:short_id]", meta is the type's meta, short_id
is the leftmost 8 digits of type's id. For builtins, ignore colon and short_id.
This property can be overrided with other type.
pointer - boolean.
By default, member is an instance defined by its type.
When this property is set to true, member is a pointer of its type.
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "bandwidth", "pointer": true}
]
} -->
Then type of the member 'bandwidth' will be changed to pointer, as below:
virBandwidthDefPtr bandwidth;
specified - boolean.
When this property is set to true, member will be attached with a new member
which specifies the existence of its master. The new additional member's form
is "bool XXX_specified;".
For example, inject directive as below:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "managerid", "name": "managerID", "specified": true},
... ...
]
} -->
<element name="parameters">
... ...
An addional member 'managerID_specified' will follow the 'managerID'.
uint8_t managerID;
bool managerID_specified;
declare.comment - string.
Specify a comment behind member's declaration in struct.
For example, inject a directive like below:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "uid", "declare.comment": "exempt from syntax-check"}
]
} -->
<element name="zpci">
... ...
Then the declaration of the member 'uid' is like below:
int uid; /* exempt from syntax-check */
parse.disable - boolean.
When this property is set to true, member will not appear in parsefunc.
parse.args - array. Only for struct-type member.
Specify the extra actual arguments when member is parsed by calling parsefunc.
If member's type is struct and the struct's parsefunc has extra formal
argument, passing actual arguments by this property.
For example, type of the member 'host' is virNetworkDHCPHostDef, and the parsefunc
related to the type has an extra arg partialOkey(specified by "args" in "parsefunc").
Here this extra arg needs to be specified a value.
Directive is like below:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "host", "parse.args": [{"name": "partialOkay", "value": "false"}]}
]
} -->
<element name="dhcp">
... ...
Then the parsing code for 'host' in the parsefunc is like below:
if (virNetworkIPDHCPHostDefParseXML(node, &def->hosts[i], ctxt, false) < 0)
goto error;
A specific value is passed into the function as an actual arg.
parse.instname - string. Only for struct-type member.
When member's type is struct and the struct's parsefunc needs a instance-name
argument, passing instance name through this property.
For example:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "ip", "parse.instname": "def->name"}
]
} -->
<element name="network">
... ...
Then parsing code for 'ip' in the parsefunc will be:
if (virNetworkIPDefParseXML(node, &def->ips[i], ctxt, def->name) < 0)
goto error;
The 'def->name' is passed in as instance-name.
parse.default - string.
Specify a value for member when it doesn't get value from XML.
For example:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "mode", "parse.default": "VIR_NETWORK_FORWARD_NAT"}
]
} -->
Then when the member 'mode' dosen't get value in parsefunc, it will be
assigned the default value 'VIR_NETWORK_FORWARD_NAT'.
format.disable - boolean.
When this property is set to true, member will not be skipped in formatfunc.
format.nocheck - boolean.
Format and dump member to XML despite of the conditions.
By default, a member needs to be check whether its value is empty before
handling it in formatfunc. When this property is set to true, ignore these
checks and always output.
For example:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "domain", "format.nocheck": true}
]
} -->
<define name="pciaddress">
... ...
Then whether or not the member 'domain' is zero, it will be formatted and output.
format.precheck - string. Only valid when format.nocheck isn't set.
In formatfunc, invoke a function as condition-check rather than default checks.
The function should be hardcoded in advance and its name is specified by this
property.
For example, in virNetworkDefFormatBuf, the member 'connections' will be formatted
and output only when some conditions evaluate to true. So set this property to
enable a check function named 'checkXMLInactive', and put logic of checking
conditions into the function.
For example:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "connections", "format.precheck": "checkXMLInactive"}
]
} -->
<define name="network">
... ...
Then RNG2C generates the declaration of checkXMLInactive.
bool
checkXMLInactive(const unsigned int def G_GNUC_UNUSED,
const virNetworkDef *parent G_GNUC_UNUSED,
unsigned int flags);
Implement this function in libvirt source.
format.fmt - string.
For every builtin, there's a default format when they're formatting.
This property can be overrided to change the default format.
For example:
<!-- VIRT:DIRECTIVE {
"members": [
{"id": "slot", "format.fmt": "0x%02x"}
]
} -->
<define name="pciaddress">
... ...
Then the member 'slot' will be formatting based on format "0x%02x".
format.args - array. Only for struct-type member.
This property is just like parse.args, except that it is for formatfunc.
Now it hasn't been used.
hint - string. READ-ONLY.
Display special cases for member.
The 'unpack' indicates member's type is unpack; 'pack' indicates member's type
is pack; 'union' indicates member's type is union; 'new' means this member
is added.
7) output property
This property is for structure, clearfunc, parsefunc and formatfunc.
It has two effects:
First, it controls whether or not to output the code to the files.
Second, it controls what file to output.
Value of the property indicates a base path, and it is suffixed with
".generated.[c|h]" to form two generated files.
The base path should be according to the file in which the code is about to
be replaced with generated code.
So the relationship between original files and generated files is as below:
original.c ---> original.h ---------
^ |
| v
original.generated.c original.generated.h
The original.generated.c includes original.h automatically, and the
original.h needs to be inserted with a line '#include "original.generated.h"'
manually.
This method can solve the Problem4.
Steps of each patch
=====================
Replacing hardcoded code with generated stuff, we need a series of patches.
Each patch just replaces a target, i.e. a structure or a function.
The steps are as below:
1) Execute subcommand 'list' to query the target's id and location.
2) Execute subcommand 'show' to preview the target's directive and generated code.
3) Modify the directive and re-inject it into the relax-ng file to bridge
the gap between current code and generated code.
4) Execute subcommand 'show' again to check the new directive and new generated code,
until generated code can take the place of hardcoded stuff.
5) Remove the hardcoded code.
6) Compile & Test the libvirt source.
About the series of patches
============================
Now I select network as a test target and make a try to generate
virNetworkDef, virNetworkDefClear, virNetworkDefParseXML and
virNetworkDefFormatBuf and their subordinate stuff automatically.
But this needs around 130 patches. That's too many!
So this series just include about 30 patches for evaluation.
Thanks!
Shi Lei (29):
rng2c: Add tool RNG2C to translate relax-ng into c-language-code.
maint: Call RNG2C automatically when relax-ng files change
util: Add two helper functions virXMLChildNode and virXMLChildNodeSet
util: Move VIR_ENUM_IMPL and VIR_ENUM_DECL from virenum.h to
internal.h
util: Replace virTristateBool|virTristateSwitch(hardcoded) with
namesakes(generated)
util: Add a helper function virStringParseOnOff
util: Add some helper functions for virSocketAddr
util: Add a helper function virStrToLong_u8p
conf: Replace virNetworkDNSTxtDef(hardcoded) with namesake(generated)
conf: Replace virNetworkDNSSrvDef(hardcoded) with namesake(generated)
conf: Replace virNetworkDNSHostDef(hardcoded) with namesake(generated)
conf: Replace virNetworkDNSForwarder(hardcoded) with
namesake(generated)
conf: Replace virNetworkDNSDef(hardcoded) with namesake(generated)
conf: Replace virNetworkDNSDefClear(hardcoded) with
namesake(generated)
conf: Extract error-checking code from virNetworkDNSHostDefParseXML
conf: Replace virNetworkDNSHostDefParseXML(hardcoded) with
namesake(generated)
conf: Extract error-checking code from virNetworkDNSSrvDefParseXML
conf: Replace virNetworkDNSSrvDefParseXML(hardcoded) with
namesake(generated)
conf: Extract error-checking code from virNetworkDNSTxtDefParseXML
conf: Replace virNetworkDNSTxtDefParseXML(hardcoded) with
namesake(generated)
conf: Add virNetworkDNSForwarderParseXML and
virNetworkDNSForwarderParseXMLPost
conf: Replace virNetworkDNSForwarderParseXML(hardcoded) with
namesake(generated)
conf: Extract error-checking code from virNetworkDNSDefParseXML
conf: Replace virNetworkDNSDefParseXML(hardcoded) with
namesake(generated)
conf: Apply virNetworkDNSForwarderFormatBuf(generated) in
virNetworkDNSDefFormat
conf: Apply virNetworkDNSTxtDefFormatBuf(generated) in
virNetworkDNSDefFormat
conf: Apply virNetworkDNSSrvDefFormatBuf(generated) in
virNetworkDNSDefFormat
conf: Apply virNetworkDNSHostDefFormatBuf(generated) in
virNetworkDNSDefFormat
conf: Replace virNetworkDNSDefFormat(hardcoded) with
virNetworkDNSDefFormatBuf(generated)
docs/schemas/basictypes.rng | 15 +
docs/schemas/network.rng | 76 ++
docs/schemas/networkcommon.rng | 2 +-
po/POTFILES.in | 2 +
rng2c/directive.py | 1693 +++++++++++++++++++++++++++++++
rng2c/generator.py | 504 +++++++++
rng2c/go | 8 +
rng2c/schema.json | 113 +++
rng2c/utils.py | 163 +++
src/Makefile.am | 16 +-
src/access/Makefile.inc.am | 2 +-
src/conf/Makefile.inc.am | 12 +-
src/conf/network_conf.c | 739 ++++----------
src/conf/network_conf.h | 51 +-
src/esx/Makefile.inc.am | 2 +-
src/interface/Makefile.inc.am | 2 +-
src/internal.h | 19 +
src/lxc/Makefile.inc.am | 1 +
src/network/Makefile.inc.am | 2 +-
src/network/bridge_driver.c | 2 +-
src/node_device/Makefile.inc.am | 2 +-
src/nwfilter/Makefile.inc.am | 2 +-
src/qemu/Makefile.inc.am | 1 +
src/remote/Makefile.inc.am | 2 +-
src/secret/Makefile.inc.am | 2 +-
src/storage/Makefile.inc.am | 2 +-
src/test/Makefile.inc.am | 2 +-
src/util/Makefile.inc.am | 13 +-
src/util/virenum.c | 15 -
src/util/virenum.h | 39 +-
src/util/virsocketaddr.c | 27 +
src/util/virsocketaddr.h | 10 +
src/util/virstring.c | 37 +
src/util/virstring.h | 10 +
src/util/virxml.c | 57 ++
src/util/virxml.h | 3 +
src/vbox/Makefile.inc.am | 2 +-
tests/Makefile.am | 2 +
tools/Makefile.am | 2 +
39 files changed, 3000 insertions(+), 654 deletions(-)
create mode 100644 rng2c/directive.py
create mode 100755 rng2c/generator.py
create mode 100755 rng2c/go
create mode 100644 rng2c/schema.json
create mode 100644 rng2c/utils.py
--
2.17.1
4 years, 7 months
- 4
- 38
[libvirt] [python] WIP-FYI: mypy annotations for libvirt-python
by Philipp Hahn
Hello,
Maybe you already have heads about mypy <http://mypy-lang.org/>, which
"is an experimental optional static type checker for Python that aims to
combine the benefits of dynamic (or "duck") typing and static typing".
I started to write a manual annotation file for the Python binding of
libvirt. I've attached my current version, so others can benefit from
it, too. It is far from complete, but it already helped my to find some
errors in my code.
(My latest version is also available at
<https://github.com/univention/typeshed/blob/libvirt/third_party/2and3/lib...>)
Long-term it probably would be better to teach the Python binding
"generator.py" to add the type information (PEP 484
<https://www.python.org/dev/peps/pep-0484/>) directly into the generated
"libvirt.py" file, but that's for another day.
If someone else is interested in helping with that, please feel free to
get in contact.
Philipp
--
Philipp Hahn
Open Source Software Engineer
Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
Tel.: +49 421 22232-0
Fax : +49 421 22232-99
hahn(a)univention.de
http://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
4 years, 7 months
- 2
- 2
[libvirt PATCH 0/2] gitlab: Clean up and improve ccache usage
by Andrea Bolognani
Test pipeline: https://gitlab.com/abologna/libvirt/pipelines/131128028
Andrea Bolognani (2):
gitlab: Don't define $MAKE
gitlab: Enable improved ccache usage
.gitlab-ci.yml | 13 ++++---------
1 file changed, 4 insertions(+), 9 deletions(-)
--
2.25.1
4 years, 7 months
- 2
- 6
[libvirt PATCH 00/39] Distinguish Cascadelake-Server from Skylake-Server
by Jiri Denemark
The signatures of these two CPU model differ only in stepping as both
report family 6 and model 85. Skylake-Server uses stepping 4 or less and
Cascadelake-Server uses stepping 5..7.
https://bugzilla.redhat.com/show_bug.cgi?id=1761678
Jiri Denemark (39):
cpu_x86: Drop noTSX hint for incompatible CPUs
cpu_x86: Use glib allocation for virCPU{,x86}Data
cpu_x86: Use glib allocation for virCPUx86Vendor
cpu_x86: Use glib allocation for virCPUx86Feature
cpu_x86: Use glib allocation for virCPUx86Model
cpu_x86: Use glib allocation for virCPUx86Map
cpu_x86: Use glib allocation in virCPUx86GetModels
cpu_x86: Use g_auto* in x86DataToCPU
cpu_x86: Use g_auto* in x86VendorParse
cpu_x86: Use g_auto* in x86FeatureParse
cpu_x86: Use g_auto* in x86ModelFromCPU
cpu_x86: Use g_auto* in x86ModelParse
cpu_x86: Use g_auto* in virCPUx86LoadMap
cpu_x86: Use g_auto* in virCPUx86DataParse
cpu_x86: Use g_auto* in x86Compute
cpu_x86: Use g_auto* in virCPUx86Compare
cpu_x86: Use g_auto* in x86Decode
cpu_x86: Use g_auto* in x86EncodePolicy
cpu_x86: Use g_auto* in x86Encode
cpu_x86: Use g_auto* in virCPUx86CheckFeature
cpu_x86: Use g_auto* in virCPUx86GetHost
cpu_x86: Use g_auto* in virCPUx86Baseline
cpu_x86: Use g_auto* in x86UpdateHostModel
cpu_x86: Use g_auto* in virCPUx86Update
cpu_x86: Use g_auto* in virCPUx86UpdateLive
cpu_x86: Use g_auto* in virCPUx86Translate
cpu_x86: Use g_auto* in virCPUx86ExpandFeatures
cpu_x86: Use g_auto* in virCPUx86CopyMigratable
cpu_x86: Move and rename x86ModelCopySignatures
cpu_x86: Move and rename x86ModelHasSignature
cpu_x86: Move and rename x86FormatSignatures
cpu_x86: Introduce virCPUx86SignaturesFree
cpu_x86: Introduce virCPUx86SignatureFromCPUID
cpu_x86: Replace 32b signatures in virCPUx86Model with a struct
cpu_x86: Don't check return value of x86ModelCopy
cpu_x86: Add support for stepping part of CPU signature
cputest: Add data for Intel(R) Xeon(R) Platinum 9242 CPU
cputest: Add data for Intel(R) Xeon(R) Gold 6130 CPU
cpu_map: Distinguish Cascadelake-Server from Skylake-Server
src/cpu/cpu_x86.c | 936 +++++------
src/cpu_map/x86_Cascadelake-Server-noTSX.xml | 2 +-
src/cpu_map/x86_Cascadelake-Server.xml | 2 +-
src/cpu_map/x86_Skylake-Server-IBRS.xml | 2 +-
src/cpu_map/x86_Skylake-Server-noTSX-IBRS.xml | 2 +-
src/cpu_map/x86_Skylake-Server.xml | 2 +-
tests/cputest.c | 2 +
.../x86_64-cpuid-Xeon-Gold-6130-disabled.xml | 7 +
.../x86_64-cpuid-Xeon-Gold-6130-enabled.xml | 9 +
.../x86_64-cpuid-Xeon-Gold-6130-guest.xml | 34 +
.../x86_64-cpuid-Xeon-Gold-6130-host.xml | 35 +
.../x86_64-cpuid-Xeon-Gold-6130-json.xml | 17 +
.../x86_64-cpuid-Xeon-Gold-6130.json | 1201 ++++++++++++++
.../x86_64-cpuid-Xeon-Gold-6130.sig | 4 +
.../x86_64-cpuid-Xeon-Gold-6130.xml | 54 +
...6_64-cpuid-Xeon-Platinum-9242-disabled.xml | 7 +
...86_64-cpuid-Xeon-Platinum-9242-enabled.xml | 10 +
.../x86_64-cpuid-Xeon-Platinum-9242-guest.xml | 38 +
.../x86_64-cpuid-Xeon-Platinum-9242-host.xml | 39 +
.../x86_64-cpuid-Xeon-Platinum-9242-json.xml | 21 +
.../x86_64-cpuid-Xeon-Platinum-9242.json | 1405 +++++++++++++++++
.../x86_64-cpuid-Xeon-Platinum-9242.sig | 4 +
.../x86_64-cpuid-Xeon-Platinum-9242.xml | 68 +
23 files changed, 3372 insertions(+), 529 deletions(-)
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Gold-6130-disabled.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Gold-6130-enabled.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Gold-6130-guest.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Gold-6130-host.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Gold-6130-json.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Gold-6130.json
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Gold-6130.sig
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Gold-6130.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Platinum-9242-disabled.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Platinum-9242-enabled.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Platinum-9242-guest.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Platinum-9242-host.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Platinum-9242-json.xml
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Platinum-9242.json
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Platinum-9242.sig
create mode 100644 tests/cputestdata/x86_64-cpuid-Xeon-Platinum-9242.xml
--
2.26.0
4 years, 7 months
- 2
- 81
[libvirt] [PATCH v2 1/1] qemu: hide details of fake reboot
by Nikolay Shirokovskiy
If we use fake reboot then domain goes thru running->shutdown->running
state changes with shutdown state only for short period of time. At
least this is implementation details leaking into API. And also there is
one real case when this is not convinient. I'm doing a backup with the
help of temporary block snapshot (with the help of qemu's API which is
used in the newly created libvirt's backup API). If guest is shutdowned
I want to continue to backup so I don't kill the process and domain is
in shutdown state. Later when backup is finished I want to destroy qemu
process. So I check if it is in shutdowned state and destroy it if it
is. Now if instead of shutdown domain got fake reboot then I can destroy
process in the middle of fake reboot process.
After shutdown event we also get stop event and now as domain state is
running it will be transitioned to paused state and back to running
later. Though this is not critical for the described case I guess it is
better not to leak these details to user too. So let's leave domain in
running state on stop event if fake reboot is in process.
Reconnection code handles this patch without modification. It detects
that qemu is not running due to shutdown and then calls qemuProcessShutdownOrReboot
which reboots as fake reboot flag is set.
Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy(a)virtuozzo.com>
---
Changes from v1[1]:
- rebase on current master
- add comments
- use special flag to check if we should go paused or not*
- add notes about reconnection to commit message
* Using just fake reboot flag is not reliable. What if ACPI shutdown is
ignored by guest? Reboot flag will remain set and now domain state
will remain running on plain pause.
[1] https://www.redhat.com/archives/libvir-list/2019-October/msg01827.html
src/qemu/qemu_domain.h | 1 +
src/qemu/qemu_process.c | 61 ++++++++++++++++++++++++-----------------
2 files changed, 37 insertions(+), 25 deletions(-)
diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
index a32852047c..a39b9546ae 100644
--- a/src/qemu/qemu_domain.h
+++ b/src/qemu/qemu_domain.h
@@ -319,6 +319,7 @@ struct _qemuDomainObjPrivate {
char *lockState;
bool fakeReboot;
+ bool pausedShutdown;
virTristateBool allowReboot;
int jobs_queued;
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index 7e1db50e8f..3e5fe3b6de 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -505,6 +505,7 @@ qemuProcessFakeReboot(void *opaque)
qemuDomainObjEndJob(driver, vm);
cleanup:
+ priv->pausedShutdown = false;
if (ret == -1)
ignore_value(qemuProcessKill(vm, VIR_QEMU_PROCESS_KILL_FORCE));
virDomainObjEndAPI(&vm);
@@ -528,6 +529,7 @@ qemuProcessShutdownOrReboot(virQEMUDriverPtr driver,
vm) < 0) {
VIR_ERROR(_("Failed to create reboot thread, killing domain"));
ignore_value(qemuProcessKill(vm, VIR_QEMU_PROCESS_KILL_NOWAIT));
+ priv->pausedShutdown = false;
virObjectUnref(vm);
}
} else {
@@ -589,35 +591,41 @@ qemuProcessHandleShutdown(qemuMonitorPtr mon G_GNUC_UNUSED,
goto unlock;
}
- VIR_DEBUG("Transitioned guest %s to shutdown state",
- vm->def->name);
- virDomainObjSetState(vm,
- VIR_DOMAIN_SHUTDOWN,
- VIR_DOMAIN_SHUTDOWN_UNKNOWN);
+ /* In case of fake reboot qemu shutdown state is transient so don't
+ * change domain state nor send events. */
+ if (!priv->fakeReboot) {
+ VIR_DEBUG("Transitioned guest %s to shutdown state",
+ vm->def->name);
+ virDomainObjSetState(vm,
+ VIR_DOMAIN_SHUTDOWN,
+ VIR_DOMAIN_SHUTDOWN_UNKNOWN);
- switch (guest_initiated) {
- case VIR_TRISTATE_BOOL_YES:
- detail = VIR_DOMAIN_EVENT_SHUTDOWN_GUEST;
- break;
+ switch (guest_initiated) {
+ case VIR_TRISTATE_BOOL_YES:
+ detail = VIR_DOMAIN_EVENT_SHUTDOWN_GUEST;
+ break;
- case VIR_TRISTATE_BOOL_NO:
- detail = VIR_DOMAIN_EVENT_SHUTDOWN_HOST;
- break;
+ case VIR_TRISTATE_BOOL_NO:
+ detail = VIR_DOMAIN_EVENT_SHUTDOWN_HOST;
+ break;
- case VIR_TRISTATE_BOOL_ABSENT:
- case VIR_TRISTATE_BOOL_LAST:
- default:
- detail = VIR_DOMAIN_EVENT_SHUTDOWN_FINISHED;
- break;
- }
+ case VIR_TRISTATE_BOOL_ABSENT:
+ case VIR_TRISTATE_BOOL_LAST:
+ default:
+ detail = VIR_DOMAIN_EVENT_SHUTDOWN_FINISHED;
+ break;
+ }
- event = virDomainEventLifecycleNewFromObj(vm,
- VIR_DOMAIN_EVENT_SHUTDOWN,
- detail);
+ event = virDomainEventLifecycleNewFromObj(vm,
+ VIR_DOMAIN_EVENT_SHUTDOWN,
+ detail);
- if (virDomainObjSave(vm, driver->xmlopt, cfg->stateDir) < 0) {
- VIR_WARN("Unable to save status on vm %s after state change",
- vm->def->name);
+ if (virDomainObjSave(vm, driver->xmlopt, cfg->stateDir) < 0) {
+ VIR_WARN("Unable to save status on vm %s after state change",
+ vm->def->name);
+ }
+ } else {
+ priv->pausedShutdown = true;
}
if (priv->agent)
@@ -651,7 +659,10 @@ qemuProcessHandleStop(qemuMonitorPtr mon G_GNUC_UNUSED,
reason = priv->pausedReason;
priv->pausedReason = VIR_DOMAIN_PAUSED_UNKNOWN;
- if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING) {
+ /* In case of fake reboot qemu paused state is transient so don't
+ * reveal it in domain state nor sent events */
+ if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING &&
+ !priv->pausedShutdown) {
if (priv->job.asyncJob == QEMU_ASYNC_JOB_MIGRATION_OUT) {
if (priv->job.current->status == QEMU_DOMAIN_JOB_STATUS_POSTCOPY)
reason = VIR_DOMAIN_PAUSED_POSTCOPY;
--
2.23.0
4 years, 7 months
- 4
- 5
[virttools-planet PATCH 0/4] Switch planet.virt-tools.org over to use GitLab Pages
by Daniel P. Berrangé
This introduces use of GitLab CI + Pages to replace the current
OpenShift application which is only admin accessible by myself.
It also has automatic integration with LetsEncrypt guaranteeing
that we'll never have expired certificates.
Daniel P. Berrangé (4):
Convert README to markdown format
Introduce use of GitLab CI for publishing to GitLab Pages
Remove obsolete openshift hosting configuration
Drop Amit Shah and Nathan Gauër
.gitlab-ci.yml | 14 +
README | 39 --
README.md | 43 ++
openshift/templates/.gitignore | 2 -
openshift/templates/update-tls.sh | 16 -
openshift/templates/virttools-planet-tls.json | 82 ----
openshift/templates/virttools-planet.json | 425 ------------------
updater/planet-cache.py => planet-cache.py | 0
updater/planet.py => planet.py | 0
{updater/planet => planet}/__init__.py | 0
{updater/planet => planet}/atomstyler.py | 0
{updater/planet => planet}/cache.py | 0
.../compat_logging/__init__.py | 0
.../compat_logging/config.py | 0
.../compat_logging/handlers.py | 0
{updater/planet => planet}/feedparser.py | 0
{updater/planet => planet}/htmltmpl.py | 0
{updater/planet => planet}/sanitize.py | 0
{updater/planet => planet}/tests/__init__.py | 0
.../planet => planet}/tests/test_channel.py | 0
{updater/planet => planet}/tests/test_main.py | 0
.../planet => planet}/tests/test_sanitize.py | 0
{updater/planet => planet}/timeoutsocket.py | 0
.../images/alexbennee.jpeg | Bin
.../virt-tools => public}/images/berrange.png | Bin
.../virt-tools => public}/images/cole.png | Bin
.../images/ehabkost.jpeg | Bin
.../images/header-bg.png | Bin
.../virt-tools => public}/images/kashyap.jpeg | Bin
.../virt-tools => public}/images/logo.png | Bin
.../virt-tools => public}/images/logo.xcf | Bin
.../virt-tools => public}/images/otubo.png | Bin
.../virt-tools => public}/images/qemu.png | Bin
.../virt-tools => public}/images/rjones.jpeg | Bin
.../virt-tools => public}/images/sgarzare.png | Bin
.../virt-tools => public}/images/teuf.png | Bin
.../images/thomashuth.png | Bin
.../virt-tools => public}/images/ybettan.png | Bin
.../virt-tools => public}/images/zeenix.png | Bin
updater/app.py | 18 -
.../virt-tools => virt-tools}/atom.xml.tmpl | 0
.../basic/index.html.tmpl | 7 +-
.../basic/style.css.tmpl | 0
{updater/virt-tools => virt-tools}/config.ini | 10 +-
.../foafroll.xml.tmpl | 0
.../virt-tools => virt-tools}/opml.xml.tmpl | 0
.../virt-tools => virt-tools}/rss10.xml.tmpl | 0
.../virt-tools => virt-tools}/rss20.xml.tmpl | 0
web/httpd-cfg/cors.conf | 3 -
49 files changed, 62 insertions(+), 597 deletions(-)
create mode 100644 .gitlab-ci.yml
delete mode 100644 README
create mode 100644 README.md
delete mode 100644 openshift/templates/.gitignore
delete mode 100755 openshift/templates/update-tls.sh
delete mode 100644 openshift/templates/virttools-planet-tls.json
delete mode 100644 openshift/templates/virttools-planet.json
rename updater/planet-cache.py => planet-cache.py (100%)
rename updater/planet.py => planet.py (100%)
rename {updater/planet => planet}/__init__.py (100%)
rename {updater/planet => planet}/atomstyler.py (100%)
rename {updater/planet => planet}/cache.py (100%)
rename {updater/planet => planet}/compat_logging/__init__.py (100%)
rename {updater/planet => planet}/compat_logging/config.py (100%)
rename {updater/planet => planet}/compat_logging/handlers.py (100%)
rename {updater/planet => planet}/feedparser.py (100%)
rename {updater/planet => planet}/htmltmpl.py (100%)
rename {updater/planet => planet}/sanitize.py (100%)
rename {updater/planet => planet}/tests/__init__.py (100%)
rename {updater/planet => planet}/tests/test_channel.py (100%)
rename {updater/planet => planet}/tests/test_main.py (100%)
rename {updater/planet => planet}/tests/test_sanitize.py (100%)
rename {updater/planet => planet}/timeoutsocket.py (100%)
rename {updater/virt-tools => public}/images/alexbennee.jpeg (100%)
rename {updater/virt-tools => public}/images/berrange.png (100%)
rename {updater/virt-tools => public}/images/cole.png (100%)
rename {updater/virt-tools => public}/images/ehabkost.jpeg (100%)
rename {updater/virt-tools => public}/images/header-bg.png (100%)
rename {updater/virt-tools => public}/images/kashyap.jpeg (100%)
rename {updater/virt-tools => public}/images/logo.png (100%)
rename {updater/virt-tools => public}/images/logo.xcf (100%)
rename {updater/virt-tools => public}/images/otubo.png (100%)
rename {updater/virt-tools => public}/images/qemu.png (100%)
rename {updater/virt-tools => public}/images/rjones.jpeg (100%)
rename {updater/virt-tools => public}/images/sgarzare.png (100%)
rename {updater/virt-tools => public}/images/teuf.png (100%)
rename {updater/virt-tools => public}/images/thomashuth.png (100%)
rename {updater/virt-tools => public}/images/ybettan.png (100%)
rename {updater/virt-tools => public}/images/zeenix.png (100%)
delete mode 100755 updater/app.py
rename {updater/virt-tools => virt-tools}/atom.xml.tmpl (100%)
rename {updater/virt-tools => virt-tools}/basic/index.html.tmpl (91%)
rename {updater/virt-tools => virt-tools}/basic/style.css.tmpl (100%)
rename {updater/virt-tools => virt-tools}/config.ini (96%)
rename {updater/virt-tools => virt-tools}/foafroll.xml.tmpl (100%)
rename {updater/virt-tools => virt-tools}/opml.xml.tmpl (100%)
rename {updater/virt-tools => virt-tools}/rss10.xml.tmpl (100%)
rename {updater/virt-tools => virt-tools}/rss20.xml.tmpl (100%)
delete mode 100644 web/httpd-cfg/cors.conf
--
2.24.1
4 years, 7 months
- 2
- 10
[libvirt-go-xml PATCH 0/2] Switch CI from Travis to GitLab
by Daniel P. Berrangé
Daniel P. Berrangé (2):
gitlab: add CI definition for GitLab
travis: delete CI configuration
.gitlab-ci.yml | 25 +++++++++++++++++++++++++
.travis.yml | 13 -------------
README.md | 2 +-
3 files changed, 26 insertions(+), 14 deletions(-)
create mode 100644 .gitlab-ci.yml
delete mode 100644 .travis.yml
--
2.24.1
4 years, 7 months
- 2
- 5