8:41 a.m.
Hello,
In response to a lot of the talk of qemud lately on qemu-devel, the
libvirt community would like to put forward a proposal to help enable
debug/advanced options when using various hypervisors. The goals of
this API are:
1) To enable more rapid access to hypervisor features before proper
libvirt API's are designed around them.
2) To facilitate debugging and access to advanced features that may
not fit into the normal libvirt world-view.
Caveats:
1) Unlike other libvirt API's, this one will explicitly *not* be
guaranteed ABI/API compatible between libvirt updates.
2) Again unlike other libvirt API's, access and configuration of
the debug section of a domain will be highly hypervisor dependent.
3) Application developers will be strongly discouraged from using
this API because of the above 2 issues. To help in this, the
API's will be in a separate library that developers will explicitly
have to link to, and it will have a different (but largely compatible)
wire protocol.
4) We don't expect this API to solve all of the issues brought up
during the qemud discussion. Our initial goal is just to give
ready access of the qemu command-line and monitor to developers.
With that being said, our initial proposal follows. We expect this
to evolve over time as we get more feedback, but we think this
proposal addresses at least 2 of the major pain points qemu developers
have while trying to use libvirt.
The initial debug XML for qemu would be:
<domain type='kvm'>
<name>myguest</name>
...
<debug>
<monitorpassthrough/>
<commandline>
<extra>qemu arguments</extra>
<alter option="optname">
<rename>newname</rename>
<match>REGEXP</match>
<modify>foo=on</modify>
<extra>-bar</extra>
</alter>
</commandline>
</debug>
</domain>
Raw access to the qemu monitor will be disabled by default; the
<monitorpassthrough/> tag enables the ability to send QMP (or
text, if you are using older qemu) messages straight through to the
monitor. To do this there will be an additional API entry point
named virDomainDebugCommand() which takes an arbitrary string
and passes it to the monitor, and returns an arbitrary string as
a result. Thus you could pass in either "info cpus" if using the
text monitor or '{ "execute": "query-cpus" }' if using QMP.
The <commandline><extra> tag does exactly what you might expect; appends
the exact string to the qemu command-line.
The <alter> tag gets more interesting. The idea is that <alter> would
allow you to modify the libvirt-generated qemu command-line in arbitrary
ways. How this would work is probably best explained with some examples:
<commandline>
<alter option="-net">
<rename>-netdev</rename>
</alter>
</commandline>
In this example, all options named -net on the qemu command-line are
renamed to -netdev.
<commandline>
<alter option="-net">
<extra>-usbtablet</extra>
</alter>
</commandline>
In this example, if (and only if) a -net option is seen, then -usbtablet is
appended to the qemu command-line.
<commandline>
<alter option="-net">
<match>\(.*name=hostnet0.*\)</match>
<modify>\1,tap</modify>
</alter>
</commandline>
This gets more complicated (but also more powerful). In this case, any -net
option where the argument *also* matches the regex in <match> will be modified to
append
the ",tap" string. Think of it as a sed expression, s/match/modify/, against
the
argument to the -net option, and it makes more sense.
We are hoping to refine this proposal based on feedback, so comments and criticisms
are welcome!
--
Chris Lalancette
9:27 a.m.
On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
Hello,
In response to a lot of the talk of qemud lately on qemu-devel, the
libvirt community would like to put forward a proposal to help enable
debug/advanced options when using various hypervisors. The goals of
this API are:
1) To enable more rapid access to hypervisor features before proper
libvirt API's are designed around them.
2) To facilitate debugging and access to advanced features that may
not fit into the normal libvirt world-view.
Caveats:
1) Unlike other libvirt API's, this one will explicitly *not* be
guaranteed ABI/API compatible between libvirt updates.
I think we'd still aim to keep the public API stable. The bit that we
can't guarentee is the interactions with QEMU & the libvirt driver.
eg, if someone was using the API to send text monitor commands to
QEMU, and the next libvirt release switched to JSON mode, that use
would break. The API would still be valid, but the way they use it
might not be. Similarly if they add some custom extra command line
argument, this could potentially conflict with an extra command line
arg a subsquent libvirt release used. eg, they used -device to add
a PCI card with a specific PCI address. Then next libvirt release
specifies this same PCI address and then you get a clash.
So in both cases the API and XML format can be reasonably guarenteed
between releases. What we can't guarentee is that usage of these
features will be reliable across releases of libvirt.
2) Again unlike other libvirt API's, access and configuration
of
the debug section of a domain will be highly hypervisor dependent.
3) Application developers will be strongly discouraged from using
this API because of the above 2 issues. To help in this, the
API's will be in a separate library that developers will explicitly
have to link to, and it will have a different (but largely compatible)
wire protocol.
In terms of wire protocol, this would still have to run over the same
existing data channel, because we don't want apps to have to open up
multiple connections. Fortunately our protocol is designed to allow
this kind of extension
struct remote_message_header {
unsigned prog; /* REMOTE_PROGRAM */
unsigned vers; /* REMOTE_PROTOCOL_VERSION */
remote_procedure proc; /* REMOTE_PROC_x */
remote_message_type type;
unsigned serial; /* Serial number of message. */
remote_message_status status;
};
We currently have a single program 'REMOTE_PROGRAM' and its associated
'vers' and 'proc' numbers. What we would be doing is to define a new
program, say QEMU_PROGRAM. The 'remote_procedure proc' numbers can thus
start again from '1' without clashing with any of the existing APIs.
All the RPC marshalling/demarshalling code should work more or less
unchanged. We just hook in a different function dispatch table in the
daemon for that program.
4) We don't expect this API to solve all of the issues brought
up
during the qemud discussion. Our initial goal is just to give
ready access of the qemu command-line and monitor to developers.
With that being said, our initial proposal follows. We expect this
to evolve over time as we get more feedback, but we think this
proposal addresses at least 2 of the major pain points qemu developers
have while trying to use libvirt.
The initial debug XML for qemu would be:
I wouldn't call them 'debug' options, since that's just one of possible
use cases for this. What we're really doing there is exposing hypervisor
specific features, so we should just be upfront about that in our
description / naming.
<domain type='kvm'>
<name>myguest</name>
...
<debug>
<monitorpassthrough/>
<commandline>
<extra>qemu arguments</extra>
<alter option="optname">
<rename>newname</rename>
<match>REGEXP</match>
<modify>foo=on</modify>
<extra>-bar</extra>
</alter>
</commandline>
</debug>
</domain>
The concept of command line & monitor is something that is QEMU specific
and thus is not suitable for the primary XML schema. IMHO, this needs to be
done as a separate schema, linked in via an XML namespace. For example
<domain type='kvm'
xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
<name>myguest</name>
...
<qemu:commandline>
<qemu:arg>-device</arg>
<qemu:arg>lsi</arg>
</qemu:commandline>
</domain>
Raw access to the qemu monitor will be disabled by default; the
<monitorpassthrough/> tag enables the ability to send QMP (or
text, if you are using older qemu) messages straight through to the
monitor. To do this there will be an additional API entry point
named virDomainDebugCommand() which takes an arbitrary string
and passes it to the monitor, and returns an arbitrary string as
a result. Thus you could pass in either "info cpus" if using the
text monitor or '{ "execute": "query-cpus" }' if using QMP.
Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
other hypervisors have different approaches for low level extension/
debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
etc. So this should really live in a separate API namespace which is
specific to a hypervisor. For example, as a header file
#include <libvirt/libvirt-qemu.h>
Containing APIs like
int virDomainQEMUInvokeMonitor(virDomainPtr dom,
const char *command,
char **reply);
typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
virDomainPtr dom,
const char *eventname,
const char *data,
void *opaque)
int virConnectQEMUDomainEventRegister(virConnectPtr conn,
virDomainPtr dom,
const char *eventname,
virDomainQEMUMonitorCallback cb,
void *opaque);
For an add-on library
libvirt-qemu.so
I don't think there's much to be gained from having an XML element to
turn on/off use of these APIs. If an app doesn't want to use them, it
can simply not link to libvirt-qemu.so
The <commandline><extra> tag does exactly what you might
expect; appends
the exact string to the qemu command-line.
Allowing many args at once in the <extra> blob means that libvirt will
need to parse & split this up into individual args which than then be
safely passed to 'exec'. It is better to specify one arg per element
to avoid this fragile parsing problem.
The <alter> tag gets more interesting. The idea is that
<alter> would
allow you to modify the libvirt-generated qemu command-line in arbitrary
ways. How this would work is probably best explained with some examples:
<commandline>
<alter option="-net">
<rename>-netdev</rename>
</alter>
</commandline>
In this example, all options named -net on the qemu command-line are
renamed to -netdev.
In this example, if (and only if) a -net option is seen, then -usbtablet is
appended to the qemu command-line.
<commandline>
<alter option="-net">
<match>\(.*name=hostnet0.*\)</match>
<modify>\1,tap</modify>
</alter>
</commandline>
This gets more complicated (but also more powerful). In this case, any -net
I think this alteration of existing args is faaaar too complex & fragile,
and way overkill. If the arg that libvirt generates isn't what someone
needs, then remove the bit of the guest config responsible for that and
add a complete extra arg, rather than munging the existing one.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
1:16 p.m.
On 04/09/2010 10:27 AM, Daniel P. Berrange wrote:
The concept of command line & monitor is something that is QEMU
specific
and thus is not suitable for the primary XML schema. IMHO, this needs to be
done as a separate schema, linked in via an XML namespace. For example
<domain type='kvm'
xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
<name>myguest</name>
...
<qemu:commandline>
<qemu:arg>-device</arg>
<qemu:arg>lsi</arg>
</qemu:commandline>
</domain>
This is a very good idea. I didn't really know about namespaces, but now
that you've pointed them out I think this is a great way to partition it up.
I'll start looking at this.
>
> Raw access to the qemu monitor will be disabled by default; the
> <monitorpassthrough/> tag enables the ability to send QMP (or
> text, if you are using older qemu) messages straight through to the
> monitor. To do this there will be an additional API entry point
> named virDomainDebugCommand() which takes an arbitrary string
> and passes it to the monitor, and returns an arbitrary string as
> a result. Thus you could pass in either "info cpus" if using the
> text monitor or '{ "execute": "query-cpus" }' if using
QMP.
Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
other hypervisors have different approaches for low level extension/
debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
etc. So this should really live in a separate API namespace which is
specific to a hypervisor. For example, as a header file
#include <libvirt/libvirt-qemu.h>
Containing APIs like
int virDomainQEMUInvokeMonitor(virDomainPtr dom,
const char *command,
char **reply);
typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
virDomainPtr dom,
const char *eventname,
const char *data,
void *opaque)
int virConnectQEMUDomainEventRegister(virConnectPtr conn,
virDomainPtr dom,
const char *eventname,
virDomainQEMUMonitorCallback cb,
void *opaque);
For an add-on library
libvirt-qemu.so
I don't think there's much to be gained from having an XML element to
turn on/off use of these APIs. If an app doesn't want to use them, it
can simply not link to libvirt-qemu.so
The reason I wanted to do this was mostly for debug/support reasons.
That is, with this element in place we can easily tell from the dumpxml
output whether a person was using the "unreliable" API's, and thus we can
tell them to try and reproduce without that in place.
> The <commandline><extra> tag does exactly what you might expect; appends
> the exact string to the qemu command-line.
Allowing many args at once in the <extra> blob means that libvirt will
need to parse & split this up into individual args which than then be
safely passed to 'exec'. It is better to specify one arg per element
to avoid this fragile parsing problem.
That is true. However, it's nicer to the user (and more like the qemu
command-line) to specify the options as one big string. In any case we
already have the code for parsing a qemu command-line (for the native-to-xml
API), so we can just adapt that.
> The <alter> tag gets more interesting. The idea is that <alter> would
> allow you to modify the libvirt-generated qemu command-line in arbitrary
> ways. How this would work is probably best explained with some examples:
>
> <commandline>
> <alter option="-net">
> <rename>-netdev</rename>
> </alter>
> </commandline>
>
> In this example, all options named -net on the qemu command-line are
> renamed to -netdev.
>
> In this example, if (and only if) a -net option is seen, then -usbtablet is
> appended to the qemu command-line.
>
> <commandline>
> <alter option="-net">
> <match>\(.*name=hostnet0.*\)</match>
> <modify>\1,tap</modify>
> </alter>
> </commandline>
>
> This gets more complicated (but also more powerful). In this case, any -net
I think this alteration of existing args is faaaar too complex & fragile,
and way overkill. If the arg that libvirt generates isn't what someone
needs, then remove the bit of the guest config responsible for that and
add a complete extra arg, rather than munging the existing one.
It is complex, I agree. I also think it is pretty powerful. But if
the need arises, we can always add it later. For now we'll go with just the
monitor passthrough and simple command-line addition, and we can evolve from
there.
--
Chris Lalancette
7:41 a.m.
On Fri, Apr 09, 2010 at 02:16:06PM -0400, Chris Lalancette wrote:
On 04/09/2010 10:27 AM, Daniel P. Berrange wrote:
>> Raw access to the qemu monitor will be disabled by default; the
>> <monitorpassthrough/> tag enables the ability to send QMP (or
>> text, if you are using older qemu) messages straight through to the
>> monitor. To do this there will be an additional API entry point
>> named virDomainDebugCommand() which takes an arbitrary string
>> and passes it to the monitor, and returns an arbitrary string as
>> a result. Thus you could pass in either "info cpus" if using the
>> text monitor or '{ "execute": "query-cpus" }' if
using QMP.
>
> Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
> other hypervisors have different approaches for low level extension/
> debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
> etc. So this should really live in a separate API namespace which is
> specific to a hypervisor. For example, as a header file
>
> #include <libvirt/libvirt-qemu.h>
>
> Containing APIs like
>
> int virDomainQEMUInvokeMonitor(virDomainPtr dom,
> const char *command,
> char **reply);
>
> typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
> virDomainPtr dom,
> const char *eventname,
> const char *data,
> void *opaque)
> int virConnectQEMUDomainEventRegister(virConnectPtr conn,
> virDomainPtr dom,
> const char *eventname,
> virDomainQEMUMonitorCallback cb,
> void *opaque);
>
>
> For an add-on library
>
> libvirt-qemu.so
>
> I don't think there's much to be gained from having an XML element to
> turn on/off use of these APIs. If an app doesn't want to use them, it
> can simply not link to libvirt-qemu.so
The reason I wanted to do this was mostly for debug/support reasons.
That is, with this element in place we can easily tell from the dumpxml
output whether a person was using the "unreliable" API's, and thus we can
tell them to try and reproduce without that in place.
That doesn't tell you whether they have actually used any API or not.
It is also inconvenient if you start a guest without it, and only later
realize you want to use the extra APIs. If we want to track the actual
usage, then the first time a direct monitor command is issued, we should
simply log a warning message.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
8:56 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/12/2010 08:41 AM, Daniel P. Berrange wrote:
>> I don't think there's much to be gained from having
an XML element to
>> turn on/off use of these APIs. If an app doesn't want to use them, it
>> can simply not link to libvirt-qemu.so
>
> The reason I wanted to do this was mostly for debug/support reasons.
> That is, with this element in place we can easily tell from the dumpxml
> output whether a person was using the "unreliable" API's, and thus we
can
> tell them to try and reproduce without that in place.
That doesn't tell you whether they have actually used any API or not.
It is also inconvenient if you start a guest without it, and only later
realize you want to use the extra APIs. If we want to track the actual
usage, then the first time a direct monitor command is issued, we should
simply log a warning message.
The problem with logging a message is that it is easy to lose it. What I'm
trying to avoid here is debugging somebody's setup for hours only to find
out that they did a "pci_del" behind libvirt's back. Maybe we can just make
the
<monitorpassthrough/> a read-only flag; it is ignored in the parsing, but
it is set by "GetXMLDesc" when it detects that the virDomainQemuInvokeMonitor
has been called.
--
Chris Lalancette
9:18 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Mon, Apr 12, 2010 at 09:56:50AM -0400, Chris Lalancette wrote:
On 04/12/2010 08:41 AM, Daniel P. Berrange wrote:
>>> I don't think there's much to be gained from having an XML element
to
>>> turn on/off use of these APIs. If an app doesn't want to use them, it
>>> can simply not link to libvirt-qemu.so
>>
>> The reason I wanted to do this was mostly for debug/support reasons.
>> That is, with this element in place we can easily tell from the dumpxml
>> output whether a person was using the "unreliable" API's, and thus
we can
>> tell them to try and reproduce without that in place.
>
> That doesn't tell you whether they have actually used any API or not.
> It is also inconvenient if you start a guest without it, and only later
> realize you want to use the extra APIs. If we want to track the actual
> usage, then the first time a direct monitor command is issued, we should
> simply log a warning message.
The problem with logging a message is that it is easy to lose it. What I'm
trying to avoid here is debugging somebody's setup for hours only to find
out that they did a "pci_del" behind libvirt's back. Maybe we can just
make the
<monitorpassthrough/> a read-only flag; it is ignored in the parsing, but
it is set by "GetXMLDesc" when it detects that the virDomainQemuInvokeMonitor
has been called.
If the logging message is at level 'WARN' it will be included in syslog
for libvirtd by default, which is basic support data always collected.
The guest XML is a description of the guest configuration, which is not
a place for ad-hoc flags related to API usage.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
4:06 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
Daniel P. Berrange wrote:
I think this alteration of existing args is faaaar too complex &
fragile,
and way overkill.
Would it not be simpler, for the target audience, for the config to
contain a one-line shell script to transform particular matched
arguments in any way that's wanted?
If the arg that libvirt generates isn't what someone
needs, then remove the bit of the guest config responsible for that and
add a complete extra arg, rather than munging the existing one.
Problem with that: You might want libvirt to think it has a device
that it's controlling, still show in the GUI, still tie up with the
rest of the stack, still do the management setup, but you just want to
change some qemu-specific parameters. For example,
cache=somethingelse for block devices, iptables=tobedefined for net
devices, usbimplementation=hack for USB devices.
-- Jamie
4:30 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/09/2010 03:06 PM, Jamie Lokier wrote:
Daniel P. Berrange wrote:
> I think this alteration of existing args is faaaar too complex & fragile,
> and way overkill.
Would it not be simpler, for the target audience, for the config to
contain a one-line shell script to transform particular matched
arguments in any way that's wanted?
Yeah, having the ability to specify an optional wrapper script, that
receives the name of the normal interpreter and all the arguments the
normal interpreter would have been given, sounds like the ultimate in
flexibility at a minimum of xml. It comes at the expense of pushing the
burden of flexibility onto the shoulders of whoever writes the wrapper
script, but that seems okay to me.
--
Eric Blake eblake(a)redhat.com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
7:05 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/09/2010 11:30 PM, Eric Blake wrote:
Yeah, having the ability to specify an optional wrapper script, that
receives the name of the normal interpreter and all the arguments the
normal interpreter would have been given, sounds like the ultimate in
flexibility at a minimum of xml.
You can also just have it piped into sed.
Paolo
3:28 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote:
Daniel P. Berrange wrote:
> I think this alteration of existing args is faaaar too complex & fragile,
> and way overkill.
Would it not be simpler, for the target audience, for the config to
contain a one-line shell script to transform particular matched
arguments in any way that's wanted?
I might be missing something but I thought you could already do this
in current libvirt. ie. Change the <emulator> element [1] to point to
your shell script:
<emulator>/usr/local/bin/my-qemu-wrapper.sh</emulator>
The wrapper should get called with the command line arguments and you
can alter them however you like.
Isn't that right, Dan?
Rich.
[1] http://libvirt.org/formatdomain.html#elementsDevices
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
5:17 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
Richard W.M. Jones wrote:
On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote:
> Daniel P. Berrange wrote:
> > I think this alteration of existing args is faaaar too complex & fragile,
> > and way overkill.
>
> Would it not be simpler, for the target audience, for the config to
> contain a one-line shell script to transform particular matched
> arguments in any way that's wanted?
I might be missing something but I thought you could already do this
in current libvirt. ie. Change the <emulator> element [1] to point to
your shell script:
<emulator>/usr/local/bin/my-qemu-wrapper.sh</emulator>
The wrapper should get called with the command line arguments and you
can alter them however you like.
Isn't that right, Dan?
That forces the shell script to be relatively complicated and
recognise all qemu arguments (including version specific ones) to skip
the ones with parameters - and to know where to look for the real
target executable is.
It's not a very friendly way to add or change parts of specific
subsystems, for example modifying a blockdev parameter, but it can be
done that way if necessary.
It's not that hard to write this for trivial extra options:
<emulator>/bin/sh -c 'qemu "$0" "$@"
-extra-flag'</emulator>
(if that works).
But for things like modifications to particular devices, wouldn't it
be better to associate "add cache=special option" to a section about
one of the blockdevs which libvirt knows about, rather than the awful
prospect of parsing libvirt's output to guess which of three -device
arguments corresponds with the second SCSI drive mentioned in the
libvirt config - which might not necessarily be the second on the
command line, even.
Parsing libvirt output and having to guess which option corresponds to
what from the libvirt config sounds very fragile and also a rather
large amount of effort for something which should be easy.
And then there's modifying how network tap devices are set up and
other networking, outside qemu, but requested by libvirt. Doing that
in <emulator> sounds like a bad idea, but script hooks are the only
sane way to make networking fit every environment and requirement.
Ideally without forcing the hook writer to reimplement the normal
network setup from scratch just to tweak it a bit. I apologise if
libvirt already provides such hooks - I haven't looked at that part of
it.
-- Jamie
3:56 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:
Richard W.M. Jones wrote:
> On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote:
> > Daniel P. Berrange wrote:
> > > I think this alteration of existing args is faaaar too complex &
fragile,
> > > and way overkill.
> >
> > Would it not be simpler, for the target audience, for the config to
> > contain a one-line shell script to transform particular matched
> > arguments in any way that's wanted?
>
> I might be missing something but I thought you could already do this
> in current libvirt. ie. Change the <emulator> element [1] to point to
> your shell script:
>
> <emulator>/usr/local/bin/my-qemu-wrapper.sh</emulator>
>
> The wrapper should get called with the command line arguments and you
> can alter them however you like.
>
> Isn't that right, Dan?
That forces the shell script to be relatively complicated and
recognise all qemu arguments (including version specific ones) to skip
the ones with parameters - and to know where to look for the real
target executable is.
It's not a very friendly way to add or change parts of specific
subsystems, for example modifying a blockdev parameter, but it can be
done that way if necessary.
It's not that hard to write this for trivial extra options:
<emulator>/bin/sh -c 'qemu "$0" "$@"
-extra-flag'</emulator>
(if that works).
That won't work because we expect the emulator to be a path to
the emulator program, so yes that has to be done in the wrapper script.
Parsing libvirt output and having to guess which option corresponds
to
what from the libvirt config sounds very fragile and also a rather
large amount of effort for something which should be easy.
it's not that easy because we change the way we build the command line
for qemu as qemu evolves. for example before and after the support for
-device there have been a lot of changes.
And then there's modifying how network tap devices are set up
and
other networking, outside qemu, but requested by libvirt. Doing that
in <emulator> sounds like a bad idea, but script hooks are the only
sane way to make networking fit every environment and requirement.
Ideally without forcing the hook writer to reimplement the normal
network setup from scratch just to tweak it a bit. I apologise if
libvirt already provides such hooks - I haven't looked at that part of
it.
Well it's very new, and I still didn't wrote the documentation for it
so that's notmal, not your fault :-)
Daniel
--
Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/
daniel(a)veillard.com | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library http://libvirt.org/
7:23 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
Daniel Veillard wrote:
On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:
> It's not that hard to write this for trivial extra options:
>
> <emulator>/bin/sh -c 'qemu "$0" "$@"
-extra-flag'</emulator>
>
> (if that works).
That won't work because we expect the emulator to be a path to
the emulator program, so yes that has to be done in the wrapper script.
Ok. Being able to pass arguments would be about infinitely more
useful, so that you don't need a separate wrapper script for every
individual guest configuration - you can pass options to the single
wrapper for each variation.
Even <emulator>/path/to/wrapper qemu-0.11</emulator> is a big improvement
over <emulator>/path/to/wrapper-guest-config-6-with-qemu-0.11</emulator> ;-)
Some simple but versatile hook ideas:
- <emulator-append-option> (no space splitting, one option, appended)
- <emulator-append-options> (space splitting multiple options)
- <emulator-prepend-option>
- <emulator-prepend-options>
- <emulator-setenv name="NAME">VALUE</emulator-setenv>
- <emulator-wrap>/path/to/script</emulator-wrap>
This last one is to call the script, but pass all the options
including the path to whichever version of qemu/kvm would get
selected by libvirt. (I presume it has some selection mechanism,
given the variety of guests which break with each version of qemu
and/or qemu-kvm and/or kvm?)
(It is most unfortunate that XML forces those long prefixes everywhere
due to lack of contextual naming, unlike less verbose config formats :-/)
> Parsing libvirt output and having to guess which option
corresponds to
> what from the libvirt config sounds very fragile and also a rather
> large amount of effort for something which should be easy.
it's not that easy because we change the way we build the command line
for qemu as qemu evolves. for example before and after the support for
-device there have been a lot of changes.
I agree, which is why it's much less fragile if libvirt provides the
individual parts to hooks/wrappers, if that's asked, rather than
forcing the wrapper to parse libvirts output and guess what libvirt
does from version to version.
For sure, before and after -device, any script which is involved with
those options will probably have to change. But that's quite unusual.
Most usually I'd expect things like the order of -device options might
change depending on how libvirt pulled together it's backing
resources, for example.
-- Jamie
8:05 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Mon, Apr 12, 2010 at 01:23:08PM +0100, Jamie Lokier wrote:
Daniel Veillard wrote:
> On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:
> > Parsing libvirt output and having to guess which option
corresponds to
> > what from the libvirt config sounds very fragile and also a rather
> > large amount of effort for something which should be easy.
>
> it's not that easy because we change the way we build the command line
> for qemu as qemu evolves. for example before and after the support for
> -device there have been a lot of changes.
I agree, which is why it's much less fragile if libvirt provides the
individual parts to hooks/wrappers, if that's asked, rather than
forcing the wrapper to parse libvirts output and guess what libvirt
does from version to version.
For sure, before and after -device, any script which is involved with
those options will probably have to change. But that's quite unusual.
I could easily see this changing again in the not too distant future.
eg, we might switch to writing out a temporary config file with all
options and just passing qemu -readonly /var/lib/libvirt/qemu/guest.cfg
to avoid the QEMU arg parsing / escaping problems. Or we might stop
using command line at all for devices, and use monitor device_add to
setup all devices. Allowing ability to append extra args to the QEMU
command line is reasonably safe, while support for hooking into existing
command line args is not.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
1:47 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/12/2010 07:23 AM, Jamie Lokier wrote:
Daniel Veillard wrote:
> On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:
>
>> It's not that hard to write this for trivial extra options:
>>
>> <emulator>/bin/sh -c 'qemu "$0" "$@"
-extra-flag'</emulator>
>>
>> (if that works).
>>
> That won't work because we expect the emulator to be a path to
> the emulator program, so yes that has to be done in the wrapper script.
>
Ok. Being able to pass arguments would be about infinitely more
useful, so that you don't need a separate wrapper script for every
individual guest configuration - you can pass options to the single
wrapper for each variation.
Even<emulator>/path/to/wrapper qemu-0.11</emulator> is a big improvement
over<emulator>/path/to/wrapper-guest-config-6-with-qemu-0.11</emulator> ;-)
Some simple but versatile hook ideas:
-<emulator-append-option> (no space splitting, one option, appended)
-<emulator-append-options> (space splitting multiple options)
-<emulator-prepend-option>
-<emulator-prepend-options>
-<emulator-setenv name="NAME">VALUE</emulator-setenv>
I'd strongly suggest not focusing on manipulating command line
arguments. I think in the not too distant future, few people will pass
command line arguments as opposed to just using configuration files.
We are very close to being able to cover 90% of current users via the
config file.
Regards,
Anthony Liguori
-<emulator-wrap>/path/to/script</emulator-wrap>
This last one is to call the script, but pass all the options
including the path to whichever version of qemu/kvm would get
selected by libvirt. (I presume it has some selection mechanism,
given the variety of guests which break with each version of qemu
and/or qemu-kvm and/or kvm?)
(It is most unfortunate that XML forces those long prefixes everywhere
due to lack of contextual naming, unlike less verbose config formats :-/)
>> Parsing libvirt output and having to guess which option corresponds to
>> what from the libvirt config sounds very fragile and also a rather
>> large amount of effort for something which should be easy.
>>
> it's not that easy because we change the way we build the command line
> for qemu as qemu evolves. for example before and after the support for
> -device there have been a lot of changes.
>
I agree, which is why it's much less fragile if libvirt provides the
individual parts to hooks/wrappers, if that's asked, rather than
forcing the wrapper to parse libvirts output and guess what libvirt
does from version to version.
For sure, before and after -device, any script which is involved with
those options will probably have to change. But that's quite unusual.
Most usually I'd expect things like the order of -device options might
change depending on how libvirt pulled together it's backing
resources, for example.
-- Jamie
1:36 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/22/10 20:47, Anthony Liguori wrote:
On 04/12/2010 07:23 AM, Jamie Lokier wrote:
> Some simple but versatile hook ideas:
>
> -<emulator-append-option> (no space splitting, one option,
> appended)
> -<emulator-append-options> (space splitting multiple options)
> -<emulator-prepend-option>
> -<emulator-prepend-options>
> -<emulator-setenv name="NAME">VALUE</emulator-setenv>
I'd strongly suggest not focusing on manipulating command line
arguments. I think in the not too distant future, few people will pass
command line arguments as opposed to just using configuration files.
We are very close to being able to cover 90% of current users via the
config file.
There will always be a place for adding command line parameters for
debugging purposes. When you try to debug a problem, the first port of
call is to look at the command line arguments libvirt uses to launch
qemu and replicate those with addons. It's a nightmare today and you end
up losing networking and other things. Having an optional debug options
thing that one can set quickly from virt-manager would make it so much
easier to retry different settings fast while replicating the setup that
is normally run.
Having to manually edit xml files for making changes is just not a
viable solution.
Speaking of which, one problem with the current XML format is that it
relies on hard-coding the path to QEMU. Instead if should rely on the
system default and provide an optional tag for debugging purposes. That
would make it easier to migrate specifications from one OS to another,
without manually having to edit the QEMU tag.
Cheers,
Jes
5:30 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Thu, Apr 22, 2010 at 01:47:55PM -0500, Anthony Liguori wrote:
On 04/12/2010 07:23 AM, Jamie Lokier wrote:
>Daniel Veillard wrote:
>
>>On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:
>>
>>>It's not that hard to write this for trivial extra options:
>>>
>>> <emulator>/bin/sh -c 'qemu "$0" "$@"
-extra-flag'</emulator>
>>>
>>>(if that works).
>>>
>> That won't work because we expect the emulator to be a path to
>>the emulator program, so yes that has to be done in the wrapper script.
>>
>Ok. Being able to pass arguments would be about infinitely more
>useful, so that you don't need a separate wrapper script for every
>individual guest configuration - you can pass options to the single
>wrapper for each variation.
>
>Even<emulator>/path/to/wrapper qemu-0.11</emulator> is a big
improvement
>over<emulator>/path/to/wrapper-guest-config-6-with-qemu-0.11</emulator>
>;-)
>
>Some simple but versatile hook ideas:
>
> -<emulator-append-option> (no space splitting, one option, appended)
> -<emulator-append-options> (space splitting multiple options)
> -<emulator-prepend-option>
> -<emulator-prepend-options>
> -<emulator-setenv name="NAME">VALUE</emulator-setenv>
>
I'd strongly suggest not focusing on manipulating command line
arguments. I think in the not too distant future, few people will pass
command line arguments as opposed to just using configuration files.
Agreed, we're not having anything in the XML for modifying command
line arguments explicitly. In tiny edge case where someone actually
needs that, QEMU already provides the '-set OPTION' command line
arg that you can use to override any qdev property.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
7:53 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:
Richard W.M. Jones wrote:
> On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote:
> > Daniel P. Berrange wrote:
> > > I think this alteration of existing args is faaaar too complex &
fragile,
> > > and way overkill.
> >
> > Would it not be simpler, for the target audience, for the config to
> > contain a one-line shell script to transform particular matched
> > arguments in any way that's wanted?
>
> I might be missing something but I thought you could already do this
> in current libvirt. ie. Change the <emulator> element [1] to point to
> your shell script:
>
> <emulator>/usr/local/bin/my-qemu-wrapper.sh</emulator>
>
> The wrapper should get called with the command line arguments and you
> can alter them however you like.
>
> Isn't that right, Dan?
That forces the shell script to be relatively complicated and
recognise all qemu arguments (including version specific ones) to skip
the ones with parameters - and to know where to look for the real
target executable is.
It's not a very friendly way to add or change parts of specific
subsystems, for example modifying a blockdev parameter, but it can be
done that way if necessary.
It's not that hard to write this for trivial extra options:
<emulator>/bin/sh -c 'qemu "$0" "$@"
-extra-flag'</emulator>
(if that works).
But for things like modifications to particular devices, wouldn't it
be better to associate "add cache=special option" to a section about
one of the blockdevs which libvirt knows about, rather than the awful
prospect of parsing libvirt's output to guess which of three -device
arguments corresponds with the second SCSI drive mentioned in the
libvirt config - which might not necessarily be the second on the
command line, even.
Every device has a unique alias in the XML, which maps onto the 'id='
parameter for QEMU so you can match up them. We need to consider the
tradeoff between what's likely to be neccessary vs cost of supporting
the extensions. Allowing addition of arbitrary extra command lines is
a trivial extension with low support cost & covers the most pressing
feature timelag problems. Munging existing dev options is a less common
scenario with significantly higher support burden which I don't really
consider worth explicitly supporting beyond the existing ability to add
a wrapper script.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
10:20 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Fri, 9 Apr 2010 15:27:17 +0100
"Daniel P. Berrange" <berrange(a)redhat.com> wrote:
On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
> Hello,
> In response to a lot of the talk of qemud lately on qemu-devel, the
> libvirt community would like to put forward a proposal to help enable
> debug/advanced options when using various hypervisors. The goals of
> this API are:
>
> 1) To enable more rapid access to hypervisor features before proper
> libvirt API's are designed around them.
> 2) To facilitate debugging and access to advanced features that may
> not fit into the normal libvirt world-view.
>
> Caveats:
> 1) Unlike other libvirt API's, this one will explicitly *not* be
> guaranteed ABI/API compatible between libvirt updates.
I think we'd still aim to keep the public API stable. The bit that we
can't guarentee is the interactions with QEMU & the libvirt driver.
eg, if someone was using the API to send text monitor commands to
QEMU, and the next libvirt release switched to JSON mode, that use
would break. The API would still be valid, but the way they use it
might not be. Similarly if they add some custom extra command line
argument, this could potentially conflict with an extra command line
arg a subsquent libvirt release used. eg, they used -device to add
a PCI card with a specific PCI address. Then next libvirt release
specifies this same PCI address and then you get a clash.
Another problem is issuing Monitor commands that could confuse libvirt's
real API. Say, adding a device libvirt doesn't know about or stopping the VM
while libvirt thinks it's still running or anything like that.
When I suggested the monitor passthrough idea, I thought it could be a
workaround for the feature time lag problem, but as stated by Avi, I'm
afraid that this can potentially cause more problems than it solves.
It's very desirable for debugging (item 2 above) and maybe to try new
things, but people couldn't use it for anything serious, which is the
major issue in this subject (item 1 above).
Now, Anthony's idea of having libqemu has the same issues, which seems
to mean that the only way to really fix the time lag is to fix the
time lag :)
> Raw access to the qemu monitor will be disabled by default; the
> <monitorpassthrough/> tag enables the ability to send QMP (or
> text, if you are using older qemu) messages straight through to the
> monitor. To do this there will be an additional API entry point
> named virDomainDebugCommand() which takes an arbitrary string
> and passes it to the monitor, and returns an arbitrary string as
> a result. Thus you could pass in either "info cpus" if using the
> text monitor or '{ "execute": "query-cpus" }' if using
QMP.
Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
other hypervisors have different approaches for low level extension/
debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
etc. So this should really live in a separate API namespace which is
specific to a hypervisor. For example, as a header file
#include <libvirt/libvirt-qemu.h>
Containing APIs like
int virDomainQEMUInvokeMonitor(virDomainPtr dom,
const char *command,
char **reply);
typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
virDomainPtr dom,
const char *eventname,
const char *data,
void *opaque)
int virConnectQEMUDomainEventRegister(virConnectPtr conn,
virDomainPtr dom,
const char *eventname,
virDomainQEMUMonitorCallback cb,
void *opaque);
What about all the wonders of json? This will force clients to
have their own json <-> text converter, maybe we could provide two
APIs, just like qemu driver works internally?
1:49 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/12/2010 10:20 AM, Luiz Capitulino wrote:
On Fri, 9 Apr 2010 15:27:17 +0100
"Daniel P. Berrange"<berrange(a)redhat.com> wrote:
> On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
>
>> Hello,
>> In response to a lot of the talk of qemud lately on qemu-devel, the
>> libvirt community would like to put forward a proposal to help enable
>> debug/advanced options when using various hypervisors. The goals of
>> this API are:
>>
>> 1) To enable more rapid access to hypervisor features before proper
>> libvirt API's are designed around them.
>> 2) To facilitate debugging and access to advanced features that may
>> not fit into the normal libvirt world-view.
>>
>> Caveats:
>> 1) Unlike other libvirt API's, this one will explicitly *not* be
>> guaranteed ABI/API compatible between libvirt updates.
>>
> I think we'd still aim to keep the public API stable. The bit that we
> can't guarentee is the interactions with QEMU& the libvirt driver.
> eg, if someone was using the API to send text monitor commands to
> QEMU, and the next libvirt release switched to JSON mode, that use
> would break. The API would still be valid, but the way they use it
> might not be. Similarly if they add some custom extra command line
> argument, this could potentially conflict with an extra command line
> arg a subsquent libvirt release used. eg, they used -device to add
> a PCI card with a specific PCI address. Then next libvirt release
> specifies this same PCI address and then you get a clash.
>
Another problem is issuing Monitor commands that could confuse libvirt's
real API. Say, adding a device libvirt doesn't know about or stopping the VM
while libvirt thinks it's still running or anything like that.
We need to make libvirt and qemu smarter.
We already face this problem today with multiple libvirt users. This is
why sophisticated management mechanisms (like LDAP) have mechanisms to
do transactions or at least a series of atomic operations.
Regards,
Anthony Liguori
When I suggested the monitor passthrough idea, I thought it could
be a
workaround for the feature time lag problem, but as stated by Avi, I'm
afraid that this can potentially cause more problems than it solves.
It's very desirable for debugging (item 2 above) and maybe to try new
things, but people couldn't use it for anything serious, which is the
major issue in this subject (item 1 above).
Now, Anthony's idea of having libqemu has the same issues, which seems
to mean that the only way to really fix the time lag is to fix the
time lag :)
>> Raw access to the qemu monitor will be disabled by default; the
>> <monitorpassthrough/> tag enables the ability to send QMP (or
>> text, if you are using older qemu) messages straight through to the
>> monitor. To do this there will be an additional API entry point
>> named virDomainDebugCommand() which takes an arbitrary string
>> and passes it to the monitor, and returns an arbitrary string as
>> a result. Thus you could pass in either "info cpus" if using the
>> text monitor or '{ "execute": "query-cpus" }' if
using QMP.
>>
> Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
> other hypervisors have different approaches for low level extension/
> debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
> etc. So this should really live in a separate API namespace which is
> specific to a hypervisor. For example, as a header file
>
> #include<libvirt/libvirt-qemu.h>
>
> Containing APIs like
>
> int virDomainQEMUInvokeMonitor(virDomainPtr dom,
> const char *command,
> char **reply);
>
> typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
> virDomainPtr dom,
> const char *eventname,
> const char *data,
> void *opaque)
> int virConnectQEMUDomainEventRegister(virConnectPtr conn,
> virDomainPtr dom,
> const char *eventname,
> virDomainQEMUMonitorCallback cb,
> void *opaque);
>
What about all the wonders of json? This will force clients to
have their own json<-> text converter, maybe we could provide two
APIs, just like qemu driver works internally?
7:48 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/22/2010 09:49 PM, Anthony Liguori wrote:
> real API. Say, adding a device libvirt doesn't know about or
stopping
> the VM
> while libvirt thinks it's still running or anything like that.
Another problem is issuing Monitor commands that could confuse
libvirt's
We need to make libvirt and qemu smarter.
We already face this problem today with multiple libvirt users. This
is why sophisticated management mechanisms (like LDAP) have mechanisms
to do transactions or at least a series of atomic operations.
And people said qmp/json was overengineered...
But seriously, transactions won't help anything. qemu maintains state,
and when you have two updaters touching a shared variable not excepting
each other to, things break, no matter how much locking there is. The
only way that separate monitors could work is if they touch completely
separate state, which is difficult to ensure if you upgrade your libvirt.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
8:48 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/23/2010 07:48 AM, Avi Kivity wrote:
On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>> real API. Say, adding a device libvirt doesn't know about or
>> stopping the VM
>> while libvirt thinks it's still running or anything like that.
> Another problem is issuing Monitor commands that could confuse
> libvirt's
>
> We need to make libvirt and qemu smarter.
>
> We already face this problem today with multiple libvirt users. This
> is why sophisticated management mechanisms (like LDAP) have
> mechanisms to do transactions or at least a series of atomic operations.
And people said qmp/json was overengineered...
But seriously, transactions won't help anything. qemu maintains
state, and when you have two updaters touching a shared variable not
excepting each other to, things break, no matter how much locking
there is.
Let's consider some concrete examples. I'm using libvirt and QMP and in
QMP, I want to hot unplug a device.
Today, I do this by listing the pci devices, and issuing a pci_del that
takes a PCI address. This is intrinsically racy though because in the
worst case scenario, in between when I enumerate pci devices and do the
pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add
within libvirt of a completely different device.
There are a few ways to solve this, the simplest being that we give
devices unique ids that are never reused and instead of pci_del taking a
pci bus address, it takes a device id. That would address this race.
You can get very far by just being clever about unique ids and
notifications. There are some cases where a true RMW may be required
but I can't really think of one off hand. The way LDAP addresses this
is that it has a batched operation and a simple set of boolean
comparison operations. This lets you execute a batched operation that
will do a RMW.
The only way that separate monitors could work is if they touch
completely separate state, which is difficult to ensure if you upgrade
your libvirt.
I don't think this is as difficult of a problem as you think it is. If
you look at Active Directory and the whole set of management tools based
on it, they certainly allow concurrent management applications. You can
certainly get into trouble still but with just some careful
considerations, you can make two management applications work together
90% of the time without much fuss on the applications part.
Regards,
Anthony Liguori
9:24 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/23/2010 04:48 PM, Anthony Liguori wrote:
On 04/23/2010 07:48 AM, Avi Kivity wrote:
> On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>> real API. Say, adding a device libvirt doesn't know about or
>>> stopping the VM
>>> while libvirt thinks it's still running or anything like that.
>> Another problem is issuing Monitor commands that could confuse
>> libvirt's
>>
>> We need to make libvirt and qemu smarter.
>>
>> We already face this problem today with multiple libvirt users.
>> This is why sophisticated management mechanisms (like LDAP) have
>> mechanisms to do transactions or at least a series of atomic
>> operations.
>
> And people said qmp/json was overengineered...
>
> But seriously, transactions won't help anything. qemu maintains
> state, and when you have two updaters touching a shared variable not
> excepting each other to, things break, no matter how much locking
> there is.
Let's consider some concrete examples. I'm using libvirt and QMP and
in QMP, I want to hot unplug a device.
Today, I do this by listing the pci devices, and issuing a pci_del
that takes a PCI address. This is intrinsically racy though because
in the worst case scenario, in between when I enumerate pci devices
and do the pci_del in QMP, in libvirt, I've done a pci_del and then a
pci_add within libvirt of a completely different device.
Obviously you should do the pci_del through libvirt. Once libvirt
supports an API, use it.
There are a few ways to solve this, the simplest being that we give
devices unique ids that are never reused and instead of pci_del taking
a pci bus address, it takes a device id. That would address this race.
You can get very far by just being clever about unique ids and
notifications. There are some cases where a true RMW may be required
but I can't really think of one off hand. The way LDAP addresses this
is that it has a batched operation and a simple set of boolean
comparison operations. This lets you execute a batched operation that
will do a RMW.
I'm sure we can be very clever, but I'd rather direct this cleverness to
qemu core issues, not to the QMP (which in turn requires that users be
clever to use it correctly). QMP is a low bandwidth protocol, so races
will never show up in testing. We're laying mines here for users to
step on that we will never encounter ourselves.
> The only way that separate monitors could work is if they touch
> completely separate state, which is difficult to ensure if you
> upgrade your libvirt.
>
I don't think this is as difficult of a problem as you think it is.
If you look at Active Directory and the whole set of management tools
based on it, they certainly allow concurrent management applications.
You can certainly get into trouble still but with just some careful
considerations, you can make two management applications work together
90% of the time without much fuss on the applications part.
Maybe. We'll still have issues. For example, sVirt: if a QMP command
names a labeled resource, the non-libvirt user will have no way of
knowing how to label it.
Much better to exact a commitment from libvirt to track all QMP (and
command line) capabilities. Instead of adding cleverness to QMP, add
APIs to libvirt.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
9:36 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Fri, Apr 23, 2010 at 05:24:34PM +0300, Avi Kivity wrote:
On 04/23/2010 04:48 PM, Anthony Liguori wrote:
>On 04/23/2010 07:48 AM, Avi Kivity wrote:
>>On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>>>real API. Say, adding a device libvirt doesn't know about or
>>>>stopping the VM
>>>>while libvirt thinks it's still running or anything like that.
>>> Another problem is issuing Monitor commands that could confuse
>>>libvirt's
>>>
>>>We need to make libvirt and qemu smarter.
>>>
>>>We already face this problem today with multiple libvirt users.
>>>This is why sophisticated management mechanisms (like LDAP) have
>>>mechanisms to do transactions or at least a series of atomic
>>>operations.
>>
>>And people said qmp/json was overengineered...
>>
>>But seriously, transactions won't help anything. qemu maintains
>>state, and when you have two updaters touching a shared variable not
>>excepting each other to, things break, no matter how much locking
>>there is.
>
>Let's consider some concrete examples. I'm using libvirt and QMP and
>in QMP, I want to hot unplug a device.
>
>Today, I do this by listing the pci devices, and issuing a pci_del
>that takes a PCI address. This is intrinsically racy though because
>in the worst case scenario, in between when I enumerate pci devices
>and do the pci_del in QMP, in libvirt, I've done a pci_del and then a
>pci_add within libvirt of a completely different device.
Obviously you should do the pci_del through libvirt. Once libvirt
supports an API, use it.
>
>There are a few ways to solve this, the simplest being that we give
>devices unique ids that are never reused and instead of pci_del taking
>a pci bus address, it takes a device id. That would address this race.
>
>You can get very far by just being clever about unique ids and
>notifications. There are some cases where a true RMW may be required
>but I can't really think of one off hand. The way LDAP addresses this
>is that it has a batched operation and a simple set of boolean
>comparison operations. This lets you execute a batched operation that
>will do a RMW.
I'm sure we can be very clever, but I'd rather direct this cleverness to
qemu core issues, not to the QMP (which in turn requires that users be
clever to use it correctly). QMP is a low bandwidth protocol, so races
will never show up in testing. We're laying mines here for users to
step on that we will never encounter ourselves.
>
>> The only way that separate monitors could work is if they touch
>>completely separate state, which is difficult to ensure if you
>>upgrade your libvirt.
>>
>
>I don't think this is as difficult of a problem as you think it is.
>If you look at Active Directory and the whole set of management tools
>based on it, they certainly allow concurrent management applications.
>You can certainly get into trouble still but with just some careful
>considerations, you can make two management applications work together
>90% of the time without much fuss on the applications part.
Maybe. We'll still have issues. For example, sVirt: if a QMP command
names a labeled resource, the non-libvirt user will have no way of
knowing how to label it.
Much better to exact a commitment from libvirt to track all QMP (and
command line) capabilities. Instead of adding cleverness to QMP, add
APIs to libvirt.
Agreed. Despite adding this monitor / XML passthrough capability, we still
do not want apps to be using this at all. If there is some capability
missing that apps need then the default mode of operation is to add the
neccessary bits of libvirt. The monitor/XML pasthrough is just a short
term quick workaround until the official support is done. As such I do
not really think we need to put huge amounts of effort in the wierd
complex racey edge cases. The effort is better spent on getting the
features in libvirt.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
7:54 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
Daniel P. Berrange wrote:
> Much better to exact a commitment from libvirt to track all QMP
(and
> command line) capabilities. Instead of adding cleverness to QMP, add
> APIs to libvirt.
Agreed. Despite adding this monitor / XML passthrough capability, we still
do not want apps to be using this at all. If there is some capability
missing that apps need then the default mode of operation is to add the
neccessary bits of libvirt. The monitor/XML pasthrough is just a short
term quick workaround until the official support is done. As such I do
not really think we need to put huge amounts of effort in the wierd
complex racey edge cases. The effort is better spent on getting the
features in libvirt.
All the features? The qemu API is quite large already (look at all
the command line options and monitor commands). I'll be very
surprised if libvirt provides all of it that obscure apps may use.
I'm thinking of features which are relatively obscure but nonetheless
useful to a small number of deployments. Probably not enough to
justify the effort building data models, specifying the XML and remote
protocol and so on in libvirt.
(Unless that becomes so easily mapped to qemu's API that it's almost an
automatic thing... Which sounds like QMP, doesn't it?)
Is libvirt ever likely to go to the effort of providing all the
easily-usable API, or hooks, for:
- sending keys to a guest, driven by a timed host script?
- rebooting the guest while switching between USB touchpad and
mouse devices, because one of them is needed during an OS
install and the other is needed after?
- changing the amount of RAM available to the guest at the next
reboot, for OS install needing more memory than run time, in a
scripted fashion when building new VMs from install disk images?
- switching the guest between qemu mode and kvm mode on the next
guest reset, because qemu is faster for some things (VGA
updates) and kvm is faster for other things, so the best choice
depends on which app you need to run on that guest
- pausing a VM, making a copy, and resuming it, so as to fork it
into two VMs (literally fork)?
- setting up the host network container and NAT IP forwarding, on
demand as guests are stopped and started, so that it works in
the above scenario despite clashing IP addresses?
- running a copy of the same guest, or perhaps an entire OS
install process (scripted), many times for different qemu and
qemu-kvm versions, different BIOSes, and different
almost-equivalent hardware emulations (i.e. different NIC types,
SMP count, CPU features, disk controller type, AIO/cache type) -
for testing guests and apps on them - with some paralellism?
None of those, except perhaps the first, as what I think of as typical
virtualisation workloads, and they all seem obscure things probably
outside libvirt's remit. Probably not many users either :-)
Yet you can do them all today with qemu and scripting the monitor, and
it's getting easier with QMP.
Which is fine, qemu works, but it would be great to be able to see
those guests and interact in the basic ways through the libvirt-based
GUIs?
QMP pass-through or QMP multiple monitors seems to provide most of
that, although I can see libvirt getting a bit confused about which
devices and how much RAM the guest has installed at different times.
The bit about forking guests, I'm not sure how complicated it is to
tie in to libvirt's notion of which disk images are being used, and
hooking into it's network configuration to handle the clashing
addresses.
If those things are considered to be entirely outside libvirt's remit,
that's fine with me. Fair enough: I will continue to live with ssh
and vinagre.
I'm just raising my hand as a potential user who might like to monitor
a bunch of active and inactive guests, remotely, see how much memory
they report using, etc. launch VNC viewer from the GUI, even choose
the target host based on load and migrate on demand, while also
needing a fair bit of non-standardness and qemu-level scripting too.
Imho, that probably comes under the heading of apps using pass-through
or multiple QMP monitors, which use features that probably won't and
probably shouldn't ever be handled by libvirt itself.
-- Jamie
9:25 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 08:54 AM, Jamie Lokier wrote:
All the features? The qemu API is quite large already (look at all
the command line options and monitor commands). I'll be very
surprised if libvirt provides all of it that obscure apps may use.
I'm thinking of features which are relatively obscure but nonetheless
useful to a small number of deployments. Probably not enough to
justify the effort building data models, specifying the XML and remote
protocol and so on in libvirt.
(Unless that becomes so easily mapped to qemu's API that it's almost an
automatic thing... Which sounds like QMP, doesn't it?)
Is libvirt ever likely to go to the effort of providing all the
easily-usable API, or hooks, for:
- sending keys to a guest, driven by a timed host script?
- rebooting the guest while switching between USB touchpad and
mouse devices, because one of them is needed during an OS
install and the other is needed after?
- changing the amount of RAM available to the guest at the next
reboot, for OS install needing more memory than run time, in a
scripted fashion when building new VMs from install disk images?
- switching the guest between qemu mode and kvm mode on the next
guest reset, because qemu is faster for some things (VGA
updates) and kvm is faster for other things, so the best choice
depends on which app you need to run on that guest
- pausing a VM, making a copy, and resuming it, so as to fork it
into two VMs (literally fork)?
- setting up the host network container and NAT IP forwarding, on
demand as guests are stopped and started, so that it works in
the above scenario despite clashing IP addresses?
- running a copy of the same guest, or perhaps an entire OS
install process (scripted), many times for different qemu and
qemu-kvm versions, different BIOSes, and different
almost-equivalent hardware emulations (i.e. different NIC types,
SMP count, CPU features, disk controller type, AIO/cache type) -
for testing guests and apps on them - with some paralellism?
None of those, except perhaps the first, as what I think of as typical
virtualisation workloads, and they all seem obscure things probably
outside libvirt's remit. Probably not many users either :-)
Yet you can do them all today with qemu and scripting the monitor, and
it's getting easier with QMP.
In point of fact, you can also do some of them today more-or-less with libvirt
and some scripting. However, you are right, there are probably obscure features
that libvirt will *never* be able to entirely implement, either because they
don't fit into libvirt's world-view or because they are too obscure or difficult
to do.
However, Dan's point stands; for most features that most users want to use,
it is far better to spend effort creating a cross-hypervisor, long-term API than
to spend too much effort on individual hypervisor hacks.
<snip>
I'm just raising my hand as a potential user who might like to
monitor
a bunch of active and inactive guests, remotely, see how much memory
they report using, etc. launch VNC viewer from the GUI, even choose
the target host based on load and migrate on demand, while also
needing a fair bit of non-standardness and qemu-level scripting too.
Imho, that probably comes under the heading of apps using pass-through
or multiple QMP monitors, which use features that probably won't and
probably shouldn't ever be handled by libvirt itself.
Right, and you are probably one of the users this work targets. But in
general, for those not very familiar with virtualization/qemu, we want
to steer them far clear of this API. That goes doubly true for application
developers; we want them to be able to use a stable, long-term API and
not have to worry about the nitty-gritty details of the monitor. It's that
latter group that we want to make sure doesn't use this API.
--
Chris Lalancette
9:34 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 05:25 PM, Chris Lalancette wrote:
Right, and you are probably one of the users this work targets. But
in
general, for those not very familiar with virtualization/qemu, we want
to steer them far clear of this API. That goes doubly true for application
developers; we want them to be able to use a stable, long-term API and
not have to worry about the nitty-gritty details of the monitor. It's that
latter group that we want to make sure doesn't use this API.
With qmp, we have a stable long term API, and the nitty-gritty details
are easily hidden behind a stock json parser (unfortunately some rpc
details remain). The command line is baroque, but the libvirt xml isn't
so pretty either.
The problem is a user that starts with libvirt and outgrows its
featureset. Do we want them to fall back to qmp?
--
error compiling committee.c: too many arguments to function
9:54 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Mon, Apr 26, 2010 at 05:34:22PM +0300, Avi Kivity wrote:
On 04/26/2010 05:25 PM, Chris Lalancette wrote:
>Right, and you are probably one of the users this work targets. But in
>general, for those not very familiar with virtualization/qemu, we want
>to steer them far clear of this API. That goes doubly true for application
>developers; we want them to be able to use a stable, long-term API and
>not have to worry about the nitty-gritty details of the monitor. It's that
>latter group that we want to make sure doesn't use this API.
>
With qmp, we have a stable long term API, and the nitty-gritty details
are easily hidden behind a stock json parser (unfortunately some rpc
details remain). The command line is baroque, but the libvirt xml isn't
so pretty either.
The problem is a user that starts with libvirt and outgrows its
featureset. Do we want them to fall back to qmp?
If were QMP were to be the standard mgmt API, then you are merely inverting
the problem. The problem is now a user that starts with QMP and outgrows
its featureset.
We have two layers in the stack here & neither provides a superset of the
other. As Anthony has mentioned, this has parallels with GTK (libvirt) vs
libX11 (libQMP). Programming against GTK does not give you all the features
you'd have if you programmed against X, but most people do not need them
and this is offset by the benefits GTK gives in portability across OS. GTK
does allow them to get direct access to X if really needed, but most apps
will avoid doing that if at all possible, and when it is found to be
neccessary new APIs are added to GTK so that direct X11 access remains a
short-term hack.
We shouldn't have to frame this as a libvirt vs QMP problem. libvirt uses
QMP for its work, and apps use libvirt API for the vast majority of their
needs. If direct access to QMP is needed we can now provide that for apps
as a short term hack, until suitable APIs are added to libvirt.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
10:08 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 09:54 AM, Daniel P. Berrange wrote:
On Mon, Apr 26, 2010 at 05:34:22PM +0300, Avi Kivity wrote:
> On 04/26/2010 05:25 PM, Chris Lalancette wrote:
>
>> Right, and you are probably one of the users this work targets. But in
>> general, for those not very familiar with virtualization/qemu, we want
>> to steer them far clear of this API. That goes doubly true for application
>> developers; we want them to be able to use a stable, long-term API and
>> not have to worry about the nitty-gritty details of the monitor. It's that
>> latter group that we want to make sure doesn't use this API.
>>
>>
> With qmp, we have a stable long term API, and the nitty-gritty details
> are easily hidden behind a stock json parser (unfortunately some rpc
> details remain). The command line is baroque, but the libvirt xml isn't
> so pretty either.
>
> The problem is a user that starts with libvirt and outgrows its
> featureset. Do we want them to fall back to qmp?
>
If were QMP were to be the standard mgmt API, then you are merely inverting
the problem. The problem is now a user that starts with QMP and outgrows
its featureset.
We have two layers in the stack here& neither provides a superset of the
other. As Anthony has mentioned, this has parallels with GTK (libvirt) vs
libX11 (libQMP). Programming against GTK does not give you all the features
you'd have if you programmed against X, but most people do not need them
and this is offset by the benefits GTK gives in portability across OS. GTK
does allow them to get direct access to X if really needed, but most apps
will avoid doing that if at all possible, and when it is found to be
neccessary new APIs are added to GTK so that direct X11 access remains a
short-term hack.
We shouldn't have to frame this as a libvirt vs QMP problem. libvirt uses
QMP for its work, and apps use libvirt API for the vast majority of their
needs. If direct access to QMP is needed we can now provide that for apps
as a short term hack, until suitable APIs are added to libvirt.
I agree with you (for the most part).
The one difference here is that libgtk actually implements a full widget
kit on top of libX11. That's why people use gtk and not X11 directly.
Portability was really an after thought with gtk.
If you look at libgtk and libcairo, instead of trying to replicate the
features of cairo within gtk, gtk encourages people to use cairo
directly for anything beyond the simplest 2d operations.
Regards,
Anthony Liguori
Daniel
10:20 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Mon, Apr 26, 2010 at 10:08:33AM -0500, Anthony Liguori wrote:
On 04/26/2010 09:54 AM, Daniel P. Berrange wrote:
>On Mon, Apr 26, 2010 at 05:34:22PM +0300, Avi Kivity wrote:
>
>>On 04/26/2010 05:25 PM, Chris Lalancette wrote:
>>
>>>Right, and you are probably one of the users this work targets. But in
>>>general, for those not very familiar with virtualization/qemu, we want
>>>to steer them far clear of this API. That goes doubly true for
>>>application
>>>developers; we want them to be able to use a stable, long-term API and
>>>not have to worry about the nitty-gritty details of the monitor. It's
>>>that
>>>latter group that we want to make sure doesn't use this API.
>>>
>>>
>>With qmp, we have a stable long term API, and the nitty-gritty details
>>are easily hidden behind a stock json parser (unfortunately some rpc
>>details remain). The command line is baroque, but the libvirt xml isn't
>>so pretty either.
>>
>>The problem is a user that starts with libvirt and outgrows its
>>featureset. Do we want them to fall back to qmp?
>>
>If were QMP were to be the standard mgmt API, then you are merely inverting
>the problem. The problem is now a user that starts with QMP and outgrows
>its featureset.
>
>We have two layers in the stack here& neither provides a superset of the
>other. As Anthony has mentioned, this has parallels with GTK (libvirt) vs
>libX11 (libQMP). Programming against GTK does not give you all the features
>you'd have if you programmed against X, but most people do not need them
>and this is offset by the benefits GTK gives in portability across OS. GTK
>does allow them to get direct access to X if really needed, but most apps
>will avoid doing that if at all possible, and when it is found to be
>neccessary new APIs are added to GTK so that direct X11 access remains a
>short-term hack.
>
>We shouldn't have to frame this as a libvirt vs QMP problem. libvirt uses
>QMP for its work, and apps use libvirt API for the vast majority of their
>needs. If direct access to QMP is needed we can now provide that for apps
>as a short term hack, until suitable APIs are added to libvirt.
>
I agree with you (for the most part).
The one difference here is that libgtk actually implements a full widget
kit on top of libX11. That's why people use gtk and not X11 directly.
Portability was really an after thought with gtk.
If you look at libgtk and libcairo, instead of trying to replicate the
features of cairo within gtk, gtk encourages people to use cairo
directly for anything beyond the simplest 2d operations.
For the sake of my analogy, I'd still group cairo in with gtk, not x11. It
is really just a re-implmentation of the GDK drawing layer from GTK. It
still provides a portable higher level abstraction over underlying graphics
& rendering systems it supports. eg, again an app might do the majority of
its drawing with Cairo, but occasionally need to jump outside to use XRender
or OpenGL directly for pieces that aren't supported currently in the cairo
API.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
10:55 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 10:20 AM, Daniel P. Berrange wrote:
For the sake of my analogy, I'd still group cairo in with gtk, not x11. It
is really just a re-implmentation of the GDK drawing layer from GTK. It
still provides a portable higher level abstraction over underlying graphics
& rendering systems it supports. eg, again an app might do the majority of
its drawing with Cairo, but occasionally need to jump outside to use XRender
or OpenGL directly for pieces that aren't supported currently in the cairo
API.
Yeah, that's a fair point.
Regards,
Anthony Liguori
Regards,
Daniel
1:29 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/23/2010 09:24 AM, Avi Kivity wrote:
On 04/23/2010 04:48 PM, Anthony Liguori wrote:
> On 04/23/2010 07:48 AM, Avi Kivity wrote:
>> On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>>> real API. Say, adding a device libvirt doesn't know about or
>>>> stopping the VM
>>>> while libvirt thinks it's still running or anything like that.
>>> Another problem is issuing Monitor commands that could confuse
>>> libvirt's
>>>
>>> We need to make libvirt and qemu smarter.
>>>
>>> We already face this problem today with multiple libvirt users.
>>> This is why sophisticated management mechanisms (like LDAP) have
>>> mechanisms to do transactions or at least a series of atomic
>>> operations.
>>
>> And people said qmp/json was overengineered...
>>
>> But seriously, transactions won't help anything. qemu maintains
>> state, and when you have two updaters touching a shared variable not
>> excepting each other to, things break, no matter how much locking
>> there is.
>
> Let's consider some concrete examples. I'm using libvirt and QMP and
> in QMP, I want to hot unplug a device.
>
> Today, I do this by listing the pci devices, and issuing a pci_del
> that takes a PCI address. This is intrinsically racy though because
> in the worst case scenario, in between when I enumerate pci devices
> and do the pci_del in QMP, in libvirt, I've done a pci_del and then a
> pci_add within libvirt of a completely different device.
Obviously you should do the pci_del through libvirt. Once libvirt
supports an API, use it.
It was just an example...
>
> There are a few ways to solve this, the simplest being that we give
> devices unique ids that are never reused and instead of pci_del
> taking a pci bus address, it takes a device id. That would address
> this race.
>
> You can get very far by just being clever about unique ids and
> notifications. There are some cases where a true RMW may be required
> but I can't really think of one off hand. The way LDAP addresses
> this is that it has a batched operation and a simple set of boolean
> comparison operations. This lets you execute a batched operation
> that will do a RMW.
I'm sure we can be very clever, but I'd rather direct this cleverness
to qemu core issues, not to the QMP (which in turn requires that users
be clever to use it correctly). QMP is a low bandwidth protocol, so
races will never show up in testing. We're laying mines here for
users to step on that we will never encounter ourselves.
>
>> The only way that separate monitors could work is if they touch
>> completely separate state, which is difficult to ensure if you
>> upgrade your libvirt.
>>
>
> I don't think this is as difficult of a problem as you think it is.
> If you look at Active Directory and the whole set of management tools
> based on it, they certainly allow concurrent management
> applications. You can certainly get into trouble still but with just
> some careful considerations, you can make two management applications
> work together 90% of the time without much fuss on the applications
> part.
Maybe. We'll still have issues. For example, sVirt: if a QMP command
names a labeled resource, the non-libvirt user will have no way of
knowing how to label it.
This is orthogonal to QMP and has to do strictly with how libvirt
prepares a resource for qemu.
Much better to exact a commitment from libvirt to track all QMP (and
command line) capabilities. Instead of adding cleverness to QMP, add
APIs to libvirt.
Let's step back for a minute because I think we're missing the forest
through the trees.
We're trying to address a few distinct problems:
1) Allow libvirt users to access features of qemu that are not exposed
through libvirt
2) Provide a means for non-libvirt users to interact with qemu
3) Provide a unified and interoperable view of the world for non-libvirt
and libvirt users
For (1), we all agree that the best case scenario would be for libvirt
to support every qemu feature. I think we can also all agree though
that this is not really practical and certainly not practical for
developers since there is a development cost associated with libvirt
support (to model an API appropriately).
The new API proposed addresses (1) by allowing a user to drill down to
the QMP context. It's a good solution IMHO and I think we all agree
that there's an inherent risk to this that users will have to evaluate
on a case-by-case basis. It's a good stop-gap though.
(2) is largely addressed by QMP and a config file. I'd like to see a
nice C library, but I think a lot of other folks are happy with JSON
support in higher level languages.
(3) is the place where there are still potential challenges. I think at
the very least, our goal should be to enable conversion from (2) and (1)
to be as easy as possible. That's why I have proposed implementing a C
library for the JSON transport because we could plumb that through the
new libvirt API. This would allow a user to very quickly port an
application from QMP to libvirt. In order to do this, we need the
libvirt API to expose a dedicated monitor because we'll need to be able
to manipulate events and negotiate features.
Beyond simple porting, there's a secondary question of having
non-libvirt apps co-exist with libvirt apps. I think it's a good long
term goal, but I don't think we should worry too much about it now.
Regards,
Anthony Liguori
4:46 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/23/2010 09:29 PM, Anthony Liguori wrote:
> Maybe. We'll still have issues. For example, sVirt: if a
QMP
> command names a labeled resource, the non-libvirt user will have no
> way of knowing how to label it.
This is orthogonal to QMP and has to do strictly with how libvirt
prepares a resource for qemu.
It's not orthogonal. If you allow qmp access behind libvirt's back,
it's a problem that you will have.
> Much better to exact a commitment from libvirt to track all QMP (and
> command line) capabilities. Instead of adding cleverness to QMP, add
> APIs to libvirt.
>
Let's step back for a minute because I think we're missing the forest
through the trees.
We're trying to address a few distinct problems:
1) Allow libvirt users to access features of qemu that are not exposed
through libvirt
That's an artificial problem. If libvirt exposes all features, you
don't need to solve it.
2) Provide a means for non-libvirt users to interact with qemu
We have qmp. It doesn't do multiple guest management. I think it's
reasonable to have a qemud which does (and also does sVirt and the
zillion other things libvirt does) provided we remove them from libvirt
(long term). The only problem is that it's a lot of effort.
3) Provide a unified and interoperable view of the world for
non-libvirt and libvirt users
This problem can be solved by the non-libvirt users adopting libvirt, or
the libvirt users dropping libvirt. I don't understand why we need to
add interoperability between users who choose an interoperability
library and users who don't choose an interoperability library.
For (1), we all agree that the best case scenario would be for libvirt
to support every qemu feature. I think we can also all agree though
that this is not really practical and certainly not practical for
developers since there is a development cost associated with libvirt
support (to model an API appropriately).
All except me, perhaps.
We already have two layers of feature modeling: first, we mostly emulate
real life, not invent new features. PCI hotplug existed long before
qemu had support for it. Second, we do give some thought into how we
expose it through QMP. libvirt doesn't have to invent it again, it only
has to expose it through its lovely xml and C APIs.
The new API proposed addresses (1) by allowing a user to drill down to
the QMP context. It's a good solution IMHO and I think we all agree
that there's an inherent risk to this that users will have to evaluate
on a case-by-case basis. It's a good stop-gap though.
Agree.
(2) is largely addressed by QMP and a config file. I'd like to see a
nice C library, but I think a lot of other folks are happy with JSON
support in higher level languages.
I agree with them. C is a pretty bad choice for managing qemu (or even,
C is a pretty bad choice).
(3) is the place where there are still potential challenges. I think
at the very least, our goal should be to enable conversion from (2)
and (1) to be as easy as possible. That's why I have proposed
implementing a C library for the JSON transport because we could plumb
that through the new libvirt API. This would allow a user to very
quickly port an application from QMP to libvirt. In order to do this,
we need the libvirt API to expose a dedicated monitor because we'll
need to be able to manipulate events and negotiate features.
Most likely any application that talks QMP will hide the protocol behind
a function call interface anyway.
Beyond simple porting, there's a secondary question of having
non-libvirt apps co-exist with libvirt apps. I think it's a good long
term goal, but I don't think we should worry too much about it now.
libvirt needs to either support all but the most esoteric use cases, or
to get out of the way completely.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
10:39 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/24/2010 04:46 AM, Avi Kivity wrote:
On 04/23/2010 09:29 PM, Anthony Liguori wrote:
>> Maybe. We'll still have issues. For example, sVirt: if a QMP
>> command names a labeled resource, the non-libvirt user will have no
>> way of knowing how to label it.
>
>
> This is orthogonal to QMP and has to do strictly with how libvirt
> prepares a resource for qemu.
It's not orthogonal. If you allow qmp access behind libvirt's back,
it's a problem that you will have.
My point was, if libvirt is just exposing raw qemu features, then it
should be possible for qemu to arbitrate concurrent access. If libvirt
implements features on top of qemu, then no other third party will be
able to co-exist with those features without interacting with qemu.
It's an impossible problem for qemu to solve (arbitrating access to
state stored in a third party management app).
> 1) Allow libvirt users to access features of qemu that are not
> exposed through libvirt
That's an artificial problem. If libvirt exposes all features, you
don't need to solve it.
It won't. Otherwise, we wouldn't be having this discussion.
>
> 2) Provide a means for non-libvirt users to interact with qemu
We have qmp. It doesn't do multiple guest management. I think it's
reasonable to have a qemud which does (and also does sVirt and the
zillion other things libvirt does) provided we remove them from
libvirt (long term). The only problem is that it's a lot of effort.
It depends on what things you think are important. A lot of libvirt's
complexity is based on the fact that it uses a daemon and needs to deal
with the security implications of that. You don't need explicit
labelling if you don't use a daemon. This is really the qemu model (as
opposed to the xend model). In theory, it does support this with the
session urls but they are currently second-class citizens in libvirt.
The remote dispatch also adds a fair bit of complexity and at least for
the use-cases I'm interested in, it's not an important feature.
>
> 3) Provide a unified and interoperable view of the world for
> non-libvirt and libvirt users
This problem can be solved by the non-libvirt users adopting libvirt,
or the libvirt users dropping libvirt. I don't understand why we need
to add interoperability between users who choose an interoperability
library and users who don't choose an interoperability library.
What I'd like to avoid is user confusion. Should a user use libvirt or
libqemu? If they make a decision to use libqemu and then down the road
want to use libvirt, how hard is it to switch? Fragmentation hurts the
ecosystem and discourages good applications from existing. I think it's
our responsibility to ensure there's a good management API that exists
for qemu that we can actively recommend to our users. libvirt is very
good at typical virtualization uses of qemu but qemu is much more than
just that and has lots of advanced features.
>
> For (1), we all agree that the best case scenario would be for
> libvirt to support every qemu feature. I think we can also all agree
> though that this is not really practical and certainly not practical
> for developers since there is a development cost associated with
> libvirt support (to model an API appropriately).
All except me, perhaps.
We already have two layers of feature modeling: first, we mostly
emulate real life, not invent new features. PCI hotplug existed long
before qemu had support for it. Second, we do give some thought into
how we expose it through QMP. libvirt doesn't have to invent it
again, it only has to expose it through its lovely xml and C APIs.
That's not what the libvirt community wants to do. We're very bias.
We've made decisions about how features should be exposed and what
features should be included. We want all of those features exposed
exactly how we've implemented them because we think it's the right way.
I'm not sure there's an obvious way forward unless we decide that there
is going to be two ways to interact with qemu. One way is through the
libvirt world-view and the other is through a more qemu centric view.
The problem then becomes allowing those two models to co-exist happily
together.
The alternative is to get libvirt to just act as a thin layer to expose
qemu features directly. But honestly, what's the point of libvirt if
they did that? For the most part, the pool management is not
virtualization specific and you could have libvirt provide that
functionality without it knowing a thing about qemu.
>
> The new API proposed addresses (1) by allowing a user to drill down
> to the QMP context. It's a good solution IMHO and I think we all
> agree that there's an inherent risk to this that users will have to
> evaluate on a case-by-case basis. It's a good stop-gap though.
Agree.
>
> (2) is largely addressed by QMP and a config file. I'd like to see a
> nice C library, but I think a lot of other folks are happy with JSON
> support in higher level languages.
I agree with them. C is a pretty bad choice for managing qemu (or
even, C is a pretty bad choice).
>
> (3) is the place where there are still potential challenges. I think
> at the very least, our goal should be to enable conversion from (2)
> and (1) to be as easy as possible. That's why I have proposed
> implementing a C library for the JSON transport because we could
> plumb that through the new libvirt API. This would allow a user to
> very quickly port an application from QMP to libvirt. In order to do
> this, we need the libvirt API to expose a dedicated monitor because
> we'll need to be able to manipulate events and negotiate features.
Most likely any application that talks QMP will hide the protocol
behind a function call interface anyway.
> Beyond simple porting, there's a secondary question of having
> non-libvirt apps co-exist with libvirt apps. I think it's a good
> long term goal, but I don't think we should worry too much about it now.
libvirt needs to either support all but the most esoteric use cases,
or to get out of the way completely.
That's not where it is today or the path I think they're taking. I'm
also not the person you have to convince of this.
Regards,
Anthony Liguori
6:51 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/25/2010 06:39 AM, Anthony Liguori wrote:
On 04/24/2010 04:46 AM, Avi Kivity wrote:
> On 04/23/2010 09:29 PM, Anthony Liguori wrote:
>>> Maybe. We'll still have issues. For example, sVirt: if a QMP
>>> command names a labeled resource, the non-libvirt user will have no
>>> way of knowing how to label it.
>>
>>
>> This is orthogonal to QMP and has to do strictly with how libvirt
>> prepares a resource for qemu.
>
>
> It's not orthogonal. If you allow qmp access behind libvirt's back,
> it's a problem that you will have.
My point was, if libvirt is just exposing raw qemu features, then it
should be possible for qemu to arbitrate concurrent access. If
libvirt implements features on top of qemu, then no other third party
will be able to co-exist with those features without interacting with
qemu. It's an impossible problem for qemu to solve (arbitrating
access to state stored in a third party management app).
If libvirt implement features (like sVirt or network configuration) then
it is indeed impossible for qemu to arbitrate. If we take all those
features into qemu[d], then it becomes possible to arbitrate so long as
the libvirt and the other management app don't step on each others
toes. But that's impossible to guarantee if you upgrade your libvirt
while keeping the other app unchanged.
>> 1) Allow libvirt users to access features of qemu that are not
>> exposed through libvirt
>
> That's an artificial problem. If libvirt exposes all features, you
> don't need to solve it.
It won't. Otherwise, we wouldn't be having this discussion.
Then libvirt will fade into uselessness. A successful app using libvirt
will grow, and will have new requirements. As soon as libvirt doesn't
meet those new requirements, the app will need to talk to qemu[d]
directly. Once it does that, it may as well use qemu[d] for everything;
if you can talk QMP and generate qemu command lines, there's not much
that libvirt buys you. Even the cross-hypervisor support is not that
hard to implement, especially if you only need to satisfy your own
requirements.
>>
>> 2) Provide a means for non-libvirt users to interact with qemu
>
> We have qmp. It doesn't do multiple guest management. I think it's
> reasonable to have a qemud which does (and also does sVirt and the
> zillion other things libvirt does) provided we remove them from
> libvirt (long term). The only problem is that it's a lot of effort.
It depends on what things you think are important. A lot of libvirt's
complexity is based on the fact that it uses a daemon and needs to
deal with the security implications of that. You don't need explicit
labelling if you don't use a daemon.
I don't follow. If you have multiple guests that you want off each
other's turf you have to label their resources, either statically or
dynamically. How is it related to a daemon being present?
This is really the qemu model (as opposed to the xend model).
(and the qemud model).
In theory, it does support this with the session urls but they are
currently second-class citizens in libvirt. The remote dispatch also
adds a fair bit of complexity and at least for the use-cases I'm
interested in, it's not an important feature.
If libvirt needs a local wrapper for interesting use cases, then it has
failed. You can't have a local wrapper with the esx driver, for example.
This is off-topic, but can you detail why you don't want remote dispatch
(I assume we're talking about a multiple node deployment).
>>
>> 3) Provide a unified and interoperable view of the world for
>> non-libvirt and libvirt users
>
> This problem can be solved by the non-libvirt users adopting libvirt,
> or the libvirt users dropping libvirt. I don't understand why we
> need to add interoperability between users who choose an
> interoperability library and users who don't choose an
> interoperability library.
What I'd like to avoid is user confusion. Should a user use libvirt
or libqemu? If they make a decision to use libqemu and then down the
road want to use libvirt, how hard is it to switch? Fragmentation
hurts the ecosystem and discourages good applications from existing.
I think it's our responsibility to ensure there's a good management
API that exists for qemu that we can actively recommend to our users.
libvirt is very good at typical virtualization uses of qemu but qemu
is much more than just that and has lots of advanced features.
Every typical virtualization use will eventually grow some non-typical
requirements. If libvirt explicitly refuses to support qemu features, I
don't see how we can recommend it - even if it satisfies a user's
requirements today, what about tomorrow? what about future qemu feature,
will they be exposed or not?
If that is the case then we should develop qemud (which libvirt and
other apps can use).
(even if it isn't the case I think qemud is a good idea)
>>
>> For (1), we all agree that the best case scenario would be for
>> libvirt to support every qemu feature. I think we can also all
>> agree though that this is not really practical and certainly not
>> practical for developers since there is a development cost
>> associated with libvirt support (to model an API appropriately).
>
> All except me, perhaps.
>
> We already have two layers of feature modeling: first, we mostly
> emulate real life, not invent new features. PCI hotplug existed long
> before qemu had support for it. Second, we do give some thought into
> how we expose it through QMP. libvirt doesn't have to invent it
> again, it only has to expose it through its lovely xml and C APIs.
That's not what the libvirt community wants to do. We're very bias.
We've made decisions about how features should be exposed and what
features should be included. We want all of those features exposed
exactly how we've implemented them because we think it's the right way.
I'm not sure there's an obvious way forward unless we decide that
there is going to be two ways to interact with qemu. One way is
through the libvirt world-view and the other is through a more qemu
centric view. The problem then becomes allowing those two models to
co-exist happily together.
I don't think there's a point in managing qemu through libvirt and
directly in parallel. It means a user has to learn both APIs, and for
every operation they need to check both to see what's the best way of
exploiting the feature. There will invariably be some friction.
Layers need to stack on top of each other, not live side by side or
bypass each other.
The alternative is to get libvirt to just act as a thin layer to
expose qemu features directly. But honestly, what's the point of
libvirt if they did that?
For most hypervisors, that's exactly what libvirt does. For Xen, it
also bypasses Xend and the hypervisor's API, but it shouldn't really.
Qemu is special due to the nonexistence of qemud.
Why is sVirt implemented in libvirt? it's not the logical place for it;
rather the logical place doesn't exist.
For the most part, the pool management is not virtualization specific
and you could have libvirt provide that functionality without it
knowing a thing about qemu.
Yes.
>
>> Beyond simple porting, there's a secondary question of having
>> non-libvirt apps co-exist with libvirt apps. I think it's a good
>> long term goal, but I don't think we should worry too much about it
>> now.
>
> libvirt needs to either support all but the most esoteric use cases,
> or to get out of the way completely.
That's not where it is today or the path I think they're taking. I'm
also not the person you have to convince of this.
It's fun trying all the same. But you are the person to convince that
side-by-side management is doomed to fail, yes?
--
error compiling committee.c: too many arguments to function
8:53 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/25/2010 06:51 AM, Avi Kivity wrote:
> It depends on what things you think are important. A lot of
> libvirt's complexity is based on the fact that it uses a daemon and
> needs to deal with the security implications of that. You don't need
> explicit labelling if you don't use a daemon.
I don't follow. If you have multiple guests that you want off each
other's turf you have to label their resources, either statically or
dynamically. How is it related to a daemon being present?
Because libvirt has to perform this labelling because it loses the
original user's security context.
If you invoke qemu with the original user's credentials that launched
the guest, then you don't need to do anything special with respect to
security.
IOW, libvirt does not run guests as separate users which is why it needs
to deal with security in the first place.
> This is really the qemu model (as opposed to the xend model).
(and the qemud model).
And I've said in the past that I don't like the idea of a qemud :-)
> In theory, it does support this with the session urls but they
are
> currently second-class citizens in libvirt. The remote dispatch also
> adds a fair bit of complexity and at least for the use-cases I'm
> interested in, it's not an important feature.
If libvirt needs a local wrapper for interesting use cases, then it
has failed. You can't have a local wrapper with the esx driver, for
example.
This is off-topic, but can you detail why you don't want remote
dispatch (I assume we're talking about a multiple node deployment).
Because there are dozens of remote management APIs and then all have a
concept of agents that run on the end nodes. When fitting
virtualization management into an existing management infrastructure,
you are going to always use a local API.
If you use a remote management API, you then have to worry about
security, authorization, and authentication. Just supporting SASL
really isn't enough because there are a lot fewer actual deployments of
kerberos out there (most of them are actually Active Directory and SASL
alone is not enough to interoperate with AD).
>>>
>>> 3) Provide a unified and interoperable view of the world for
>>> non-libvirt and libvirt users
>>
>> This problem can be solved by the non-libvirt users adopting
>> libvirt, or the libvirt users dropping libvirt. I don't understand
>> why we need to add interoperability between users who choose an
>> interoperability library and users who don't choose an
>> interoperability library.
>
> What I'd like to avoid is user confusion. Should a user use libvirt
> or libqemu? If they make a decision to use libqemu and then down the
> road want to use libvirt, how hard is it to switch? Fragmentation
> hurts the ecosystem and discourages good applications from existing.
> I think it's our responsibility to ensure there's a good management
> API that exists for qemu that we can actively recommend to our
> users. libvirt is very good at typical virtualization uses of qemu
> but qemu is much more than just that and has lots of advanced features.
Every typical virtualization use will eventually grow some non-typical
requirements. If libvirt explicitly refuses to support qemu features,
I don't see how we can recommend it - even if it satisfies a user's
requirements today, what about tomorrow? what about future qemu
feature, will they be exposed or not?
If that is the case then we should develop qemud (which libvirt and
other apps can use).
(even if it isn't the case I think qemud is a good idea)
Yeah, that's where I'm at. I'd eventually like libvirt to use our
provided API and I can see where it would add value to the stack (by
doing things like storage and network management).
> That's not what the libvirt community wants to do. We're
very bias.
> We've made decisions about how features should be exposed and what
> features should be included. We want all of those features exposed
> exactly how we've implemented them because we think it's the right way.
>
> I'm not sure there's an obvious way forward unless we decide that
> there is going to be two ways to interact with qemu. One way is
> through the libvirt world-view and the other is through a more qemu
> centric view. The problem then becomes allowing those two models to
> co-exist happily together.
I don't think there's a point in managing qemu through libvirt and
directly in parallel. It means a user has to learn both APIs, and for
every operation they need to check both to see what's the best way of
exploiting the feature. There will invariably be some friction.
Layers need to stack on top of each other, not live side by side or
bypass each other.
I agree with you theoretically but practically, I think it's immensely
useful as a stop-gap.
>
> The alternative is to get libvirt to just act as a thin layer to
> expose qemu features directly. But honestly, what's the point of
> libvirt if they did that?
For most hypervisors, that's exactly what libvirt does. For Xen, it
also bypasses Xend and the hypervisor's API, but it shouldn't really.
Historically, xend was so incredibly slow (especially for frequent
statistics collection) that it was a necessity.
Qemu is special due to the nonexistence of qemud.
Why is sVirt implemented in libvirt? it's not the logical place for
it; rather the logical place doesn't exist.
sVirt is not just implemented in libvirt. libvirt implements a
mechanism to set the context of a given domain and dynamically label
it's resources to isolate it.
The reason it has to assign a context to a given domain is that all
domains are launched from the same security context (the libvirtd
context) as the original user's context (the consumer of the libvirt
API) has been lost via the domain socket interface.
If you used the /session URL, then the domain would have the security
context of whomever created the guest which means that dynamic labelling
of the resources wouldn't be necessary (you would just do static labelling).
This is certainly a more secure model and it's a feature of qemu that I
really wish didn't get lost in libvirt. Again, /session can do this too
but right now, /session really isn't usable in libvirt for qemu.
>> libvirt needs to either support all but the most esoteric use cases,
>> or to get out of the way completely.
>
> That's not where it is today or the path I think they're taking. I'm
> also not the person you have to convince of this.
>
It's fun trying all the same. But you are the person to convince that
side-by-side management is doomed to fail, yes?
The world is full of large collections of code that are all doomed to
fail :-)
Regards,
Anthony Liguori
12:56 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 04:53 AM, Anthony Liguori wrote:
On 04/25/2010 06:51 AM, Avi Kivity wrote:
>> It depends on what things you think are important. A lot of
>> libvirt's complexity is based on the fact that it uses a daemon and
>> needs to deal with the security implications of that. You don't
>> need explicit labelling if you don't use a daemon.
>
>
> I don't follow. If you have multiple guests that you want off each
> other's turf you have to label their resources, either statically or
> dynamically. How is it related to a daemon being present?
Because libvirt has to perform this labelling because it loses the
original user's security context.
If you invoke qemu with the original user's credentials that launched
the guest, then you don't need to do anything special with respect to
security.
IOW, libvirt does not run guests as separate users which is why it
needs to deal with security in the first place.
What if one user has multiple guests? isolation is still needed.
One user per guest does not satisfy some security requirements. The 'M'
in selinux stands for mandatory, which means that the entities secured
can't leak information even if they want to (scenario: G1 breaks into
qemu, chmods files, G2 breaks into qemu, reads files).
>> This is really the qemu model (as opposed to the xend model).
>
> (and the qemud model).
And I've said in the past that I don't like the idea of a qemud :-)
I must have missed it. Why not? Every other hypervisor has a central
management entity.
>> In theory, it does support this with the session urls but they are
>> currently second-class citizens in libvirt. The remote dispatch
>> also adds a fair bit of complexity and at least for the use-cases
>> I'm interested in, it's not an important feature.
>
> If libvirt needs a local wrapper for interesting use cases, then it
> has failed. You can't have a local wrapper with the esx driver, for
> example.
>
> This is off-topic, but can you detail why you don't want remote
> dispatch (I assume we're talking about a multiple node deployment).
Because there are dozens of remote management APIs and then all have a
concept of agents that run on the end nodes. When fitting
virtualization management into an existing management infrastructure,
you are going to always use a local API.
When you manage esx, do you deploy an agent? I thought it was all done
via their remote APIs.
>
> Every typical virtualization use will eventually grow some
> non-typical requirements. If libvirt explicitly refuses to support
> qemu features, I don't see how we can recommend it - even if it
> satisfies a user's requirements today, what about tomorrow? what
> about future qemu feature, will they be exposed or not?
>
> If that is the case then we should develop qemud (which libvirt and
> other apps can use).
>
> (even if it isn't the case I think qemud is a good idea)
Yeah, that's where I'm at. I'd eventually like libvirt to use our
provided API and I can see where it would add value to the stack (by
doing things like storage and network management).
We do provide an API, qmp, and libvirt uses it?
>> That's not what the libvirt community wants to do. We're very
>> bias. We've made decisions about how features should be exposed and
>> what features should be included. We want all of those features
>> exposed exactly how we've implemented them because we think it's the
>> right way.
>>
>> I'm not sure there's an obvious way forward unless we decide that
>> there is going to be two ways to interact with qemu. One way is
>> through the libvirt world-view and the other is through a more qemu
>> centric view. The problem then becomes allowing those two models to
>> co-exist happily together.
>
> I don't think there's a point in managing qemu through libvirt and
> directly in parallel. It means a user has to learn both APIs, and
> for every operation they need to check both to see what's the best
> way of exploiting the feature. There will invariably be some friction.
>
> Layers need to stack on top of each other, not live side by side or
> bypass each other.
I agree with you theoretically but practically, I think it's immensely
useful as a stop-gap.
Sure. But please lets not start being clever with transactions and
atomic operations and stuff, it has to come with a label that says, if
you're using this, then something is wrong.
>>
>> The alternative is to get libvirt to just act as a thin layer to
>> expose qemu features directly. But honestly, what's the point of
>> libvirt if they did that?
>
> For most hypervisors, that's exactly what libvirt does. For Xen, it
> also bypasses Xend and the hypervisor's API, but it shouldn't really.
Historically, xend was so incredibly slow (especially for frequent
statistics collection) that it was a necessity.
Ah, reimplement rather than fix.
> Qemu is special due to the nonexistence of qemud.
>
> Why is sVirt implemented in libvirt? it's not the logical place for
> it; rather the logical place doesn't exist.
sVirt is not just implemented in libvirt. libvirt implements a
mechanism to set the context of a given domain and dynamically label
it's resources to isolate it.
The reason it has to assign a context to a given domain is that all
domains are launched from the same security context (the libvirtd
context) as the original user's context (the consumer of the libvirt
API) has been lost via the domain socket interface.
If you used the /session URL, then the domain would have the security
context of whomever created the guest which means that dynamic
labelling of the resources wouldn't be necessary (you would just do
static labelling).
This is certainly a more secure model and it's a feature of qemu that
I really wish didn't get lost in libvirt. Again, /session can do this
too but right now, /session really isn't usable in libvirt for qemu.
That's wrong for three reasons. First, selinux is not a uid replacement
(if it was libvirt could just suid $random_user before launching qemu).
Second, a single user's guests should be protected from each other.
Third, in many deployments, the guest's owner isn't logged in to supply
the credentials, it's system management that launches the guests.
There's also the case of resources that can't be permanently chowned or
assigned a security label, like disk volumes or assignable devices.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
4:56 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
2010/4/26 Avi Kivity <avi(a)redhat.com>:
[...]
> >>>> In theory, it does support this with the
session urls but they are
>>>> currently second-class citizens in libvirt. The remote dispatch also
adds a
>>>> fair bit of complexity and at least for the use-cases I'm interested
in,
>>>> it's not an important feature.
>> >>> If libvirt needs a
local wrapper for interesting use cases, then it has
>>> failed. You can't have a local wrapper with the esx driver, for
example.
>> >>> This is off-topic, but
can you detail why you don't want remote dispatch
>>> (I assume we're talking about a multiple node deployment).
> >> Because there are dozens of remote management APIs and
then all have a
>> concept of agents that run on the end nodes. When fitting virtualization
>> management into an existing management infrastructure, you are going to
>> always use a local API.
> When you manage esx, do you deploy an agent? I thought it
was all done via
> their remote APIs.
No, you don't deploy any agents.
If you manage an ESX server using VMware tools only, then you use
VMware management tools on the client PC. This tools directly use the
SOAP based remote API provided by the ESX server.
The ESX driver in libvirt does the same and uses this remote API as
the VMware management tools. No libvirt specific agents are installed
on the ESX server and no VMware specific agents (except libvirt
itself) are installed on the client PC.
Matthias
8:14 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 12:56 AM, Avi Kivity wrote:
On 04/26/2010 04:53 AM, Anthony Liguori wrote:
> On 04/25/2010 06:51 AM, Avi Kivity wrote:
>>> It depends on what things you think are important. A lot of
>>> libvirt's complexity is based on the fact that it uses a daemon and
>>> needs to deal with the security implications of that. You don't
>>> need explicit labelling if you don't use a daemon.
>>
>>
>> I don't follow. If you have multiple guests that you want off each
>> other's turf you have to label their resources, either statically or
>> dynamically. How is it related to a daemon being present?
>
> Because libvirt has to perform this labelling because it loses the
> original user's security context.
>
> If you invoke qemu with the original user's credentials that launched
> the guest, then you don't need to do anything special with respect to
> security.
>
> IOW, libvirt does not run guests as separate users which is why it
> needs to deal with security in the first place.
What if one user has multiple guests? isolation is still needed.
Don't confuse a management application's concept of users with using
separate uid's to launch guests.
One user per guest does not satisfy some security requirements. The
'M' in selinux stands for mandatory, which means that the entities
secured can't leak information even if they want to (scenario: G1
breaks into qemu, chmods files, G2 breaks into qemu, reads files).
If you're implementing a chinese firewall policy, then yes, you want to
run each guest as a separate selinux context. Starting as separate
users and setting DAC privileges appropriately will achieve this.
But you're not always implementing that type of policy. If the guest
inherits the uid, selinux context, and namespaces of whatever launches
the guest, then you have the most flexibility from a security perspective.
How do you launch a libvirt guest in a network namespace? How do you
put it in a chroot? Today, you have to make changes to libvirt whereas
in a direct launch model, you get all of the neat security features
linux supports for free.
> And I've said in the past that I don't like the idea of a
qemud :-)
I must have missed it. Why not? Every other hypervisor has a central
management entity.
Because you end up launching all guests from a single security context.
>
>>> In theory, it does support this with the session urls but they are
>>> currently second-class citizens in libvirt. The remote dispatch
>>> also adds a fair bit of complexity and at least for the use-cases
>>> I'm interested in, it's not an important feature.
>>
>> If libvirt needs a local wrapper for interesting use cases, then it
>> has failed. You can't have a local wrapper with the esx driver, for
>> example.
>>
>> This is off-topic, but can you detail why you don't want remote
>> dispatch (I assume we're talking about a multiple node deployment).
>
> Because there are dozens of remote management APIs and then all have
> a concept of agents that run on the end nodes. When fitting
> virtualization management into an existing management infrastructure,
> you are going to always use a local API.
When you manage esx, do you deploy an agent? I thought it was all
done via their remote APIs.
Historically, people have deployed agents into the console OS. In
recent versions, ESX actually includes CIM agents by default.
>>
>> Every typical virtualization use will eventually grow some
>> non-typical requirements. If libvirt explicitly refuses to support
>> qemu features, I don't see how we can recommend it - even if it
>> satisfies a user's requirements today, what about tomorrow? what
>> about future qemu feature, will they be exposed or not?
>>
>> If that is the case then we should develop qemud (which libvirt and
>> other apps can use).
>>
>> (even if it isn't the case I think qemud is a good idea)
>
> Yeah, that's where I'm at. I'd eventually like libvirt to use our
> provided API and I can see where it would add value to the stack (by
> doing things like storage and network management).
We do provide an API, qmp, and libvirt uses it?
Yeah, but we need to support more features (like guest enumeration).
>>>
>>> The alternative is to get libvirt to just act as a thin layer to
>>> expose qemu features directly. But honestly, what's the point of
>>> libvirt if they did that?
>>
>> For most hypervisors, that's exactly what libvirt does. For Xen, it
>> also bypasses Xend and the hypervisor's API, but it shouldn't really.
>
> Historically, xend was so incredibly slow (especially for frequent
> statistics collection) that it was a necessity.
Ah, reimplement rather than fix.
There's a complicated history there.
>> Qemu is special due to the nonexistence of qemud.
>>
>> Why is sVirt implemented in libvirt? it's not the logical place for
>> it; rather the logical place doesn't exist.
>
> sVirt is not just implemented in libvirt. libvirt implements a
> mechanism to set the context of a given domain and dynamically label
> it's resources to isolate it.
>
> The reason it has to assign a context to a given domain is that all
> domains are launched from the same security context (the libvirtd
> context) as the original user's context (the consumer of the libvirt
> API) has been lost via the domain socket interface.
>
> If you used the /session URL, then the domain would have the security
> context of whomever created the guest which means that dynamic
> labelling of the resources wouldn't be necessary (you would just do
> static labelling).
>
> This is certainly a more secure model and it's a feature of qemu that
> I really wish didn't get lost in libvirt. Again, /session can do
> this too but right now, /session really isn't usable in libvirt for
> qemu.
That's wrong for three reasons. First, selinux is not a uid
replacement (if it was libvirt could just suid $random_user before
launching qemu). Second, a single user's guests should be protected
from each other. Third, in many deployments, the guest's owner isn't
logged in to supply the credentials, it's system management that
launches the guests.
(1) uid's are just one part of an applications security context.
There's an selinux context, all of the various namespaces, capabilities,
etc. If you use a daemon to launch a guest, you lose all of that unless
you have a very sophisticated api.
(2) If you want to implement a policy that only a single guest can
access a single image, you can create an SELinux policy and use static
labelling to achieve that. That's just one type of policy though.
(3) The system management application can certainly create whatever
context it wants to launch a vm from. It's comes down to who's
responsible for creating the context the guest runs under. I think
doing that at the libvirt level takes away a ton of flexibility from the
management application.
Regards,
Anthony Liguori
There's also the case of resources that can't be permanently
chowned
or assigned a security label, like disk volumes or assignable devices.
8:41 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 04:14 PM, Anthony Liguori wrote:
>>
>> IOW, libvirt does not run guests as separate users which is why it
>> needs to deal with security in the first place.
>
> What if one user has multiple guests? isolation is still needed.
Don't confuse a management application's concept of users with using
separate uid's to launch guests.
Then someone needs to manage those users. A user can't suid to any
random user. You need someone privileged to allocate the new uid and su
into it.
> One user per guest does not satisfy some security requirements. The
> 'M' in selinux stands for mandatory, which means that the entities
> secured can't leak information even if they want to (scenario: G1
> breaks into qemu, chmods files, G2 breaks into qemu, reads files).
If you're implementing a chinese firewall policy, then yes, you want
to run each guest as a separate selinux context. Starting as separate
users and setting DAC privileges appropriately will achieve this.
But you're not always implementing that type of policy. If the guest
inherits the uid, selinux context, and namespaces of whatever launches
the guest, then you have the most flexibility from a security
perspective.
How do you launch a libvirt guest in a network namespace? How do you
put it in a chroot?
You pass the namespace fd and chroot fd using SCM_RIGHTS (except you
probably can't do that).
Today, you have to make changes to libvirt whereas in a direct launch
model, you get all of the neat security features linux supports for free.
But you lose tap networking, unless you have a privileged helper. And
how is the privileged helper to authenticate the qemu calling it?
>> And I've said in the past that I don't like the idea
of a qemud :-)
>
> I must have missed it. Why not? Every other hypervisor has a
> central management entity.
Because you end up launching all guests from a single security context.
Run multiple qemuds?
But what you say makes sense. It's similar to the fork() /* do
interesting stuff */ exec() model, compared to the spawn(..., hardcoded
list of interesting stuff).
>> Yeah, that's where I'm at. I'd eventually like
libvirt to use our
>> provided API and I can see where it would add value to the stack (by
>> doing things like storage and network management).
>
> We do provide an API, qmp, and libvirt uses it?
Yeah, but we need to support more features (like guest enumeration).
What are our options?
1) qemud launches, enumerates
2) user launches, qemu registers in qemud
3) user launches, qemu registers in filesystem
4) you launched it, you enumerate it
> That's wrong for three reasons. First, selinux is not a uid
> replacement (if it was libvirt could just suid $random_user before
> launching qemu). Second, a single user's guests should be protected
> from each other. Third, in many deployments, the guest's owner isn't
> logged in to supply the credentials, it's system management that
> launches the guests.
(1) uid's are just one part of an applications security context.
There's an selinux context, all of the various namespaces,
capabilities, etc. If you use a daemon to launch a guest, you lose
all of that unless you have a very sophisticated api.
True. In a perfect world, we'd use SCM_RIGHTS to channel all of these
to libvirt or qemud.
On the other hand, users don't want to do all these things by hand.
They want management to do things for them. Self launch is very
flexible, but it's not an API, and cannot be used remotely.
We could use qemud plugins to allow the user to customize the launch
process.
(2) If you want to implement a policy that only a single guest can
access a single image, you can create an SELinux policy and use static
labelling to achieve that. That's just one type of policy though.
It's also not going to work in an environment that doesn't preserve all
security labels (like direct access to volumes; /dev is on tmpfs these
days).
(3) The system management application can certainly create whatever
context it wants to launch a vm from. It's comes down to who's
responsible for creating the context the guest runs under. I think
doing that at the libvirt level takes away a ton of flexibility from
the management application.
If you want to push the flexibility slider all the way to the right you
get bare qemu. It exposes 100% of qemu capabilities. And it's not so
bad these days. But it's not something that can be remoted.
--
error compiling committee.c: too many arguments to function
8:46 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 08:41 AM, Avi Kivity wrote:
> Today, you have to make changes to libvirt whereas in a direct
launch
> model, you get all of the neat security features linux supports for
> free.
But you lose tap networking, unless you have a privileged helper. And
how is the privileged helper to authenticate the qemu calling it?
There are a variety of ways. My original proposal used a policy file.
>>> And I've said in the past that I don't like the
idea of a qemud :-)
>>
>> I must have missed it. Why not? Every other hypervisor has a
>> central management entity.
>
> Because you end up launching all guests from a single security context.
Run multiple qemuds?
But what you say makes sense. It's similar to the fork() /* do
interesting stuff */ exec() model, compared to the spawn(...,
hardcoded list of interesting stuff).
>>> Yeah, that's where I'm at. I'd eventually like libvirt to use
our
>>> provided API and I can see where it would add value to the stack
>>> (by doing things like storage and network management).
>>
>> We do provide an API, qmp, and libvirt uses it?
>
> Yeah, but we need to support more features (like guest enumeration).
What are our options?
1) qemud launches, enumerates
2) user launches, qemu registers in qemud
3) user launches, qemu registers in filesystem
4) you launched it, you enumerate it
Both 2 and 3 are appealing to me.
> (3) The system management application can certainly create
whatever
> context it wants to launch a vm from. It's comes down to who's
> responsible for creating the context the guest runs under. I think
> doing that at the libvirt level takes away a ton of flexibility from
> the management application.
If you want to push the flexibility slider all the way to the right
you get bare qemu. It exposes 100% of qemu capabilities. And it's
not so bad these days. But it's not something that can be remoted.
As I mentioned earlier, remoting is not a very important use-case to me.
Does RHEV-M actually use the remote libvirt interface? I assume it'll
talk to vdsm via some protocol and vdsm will use the local libvirt API.
I suspect most uses of libvirt are actually local uses.
Regards,
Anthony Liguori
8:53 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 04:46 PM, Anthony Liguori wrote:
>> (3) The system management application can certainly create
whatever
>> context it wants to launch a vm from. It's comes down to who's
>> responsible for creating the context the guest runs under. I think
>> doing that at the libvirt level takes away a ton of flexibility from
>> the management application.
>
> If you want to push the flexibility slider all the way to the right
> you get bare qemu. It exposes 100% of qemu capabilities. And it's
> not so bad these days. But it's not something that can be remoted.
As I mentioned earlier, remoting is not a very important use-case to me.
Does RHEV-M actually use the remote libvirt interface? I assume it'll
talk to vdsm via some protocol and vdsm will use the local libvirt API.
Yes.
I suspect most uses of libvirt are actually local uses.
I expect the same, though I'm sure a design goal was to make use of
libvirt be reasonable through the remote API. If we aren't able to
fulfil it, much of the value of libvirt goes away.
--
error compiling committee.c: too many arguments to function
8:58 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote:
On 04/26/2010 08:41 AM, Avi Kivity wrote:
>>(3) The system management application can certainly create whatever
>>context it wants to launch a vm from. It's comes down to who's
>>responsible for creating the context the guest runs under. I think
>>doing that at the libvirt level takes away a ton of flexibility from
>>the management application.
>
>If you want to push the flexibility slider all the way to the right
>you get bare qemu. It exposes 100% of qemu capabilities. And it's
>not so bad these days. But it's not something that can be remoted.
As I mentioned earlier, remoting is not a very important use-case to me.
NB, the "remote" protocol is no different from the "local" protocol
the unprivileged app uses to talk to the privileged daemon. The only
difference is unix sockets vs tcp sockets. We essentially get the
remote access capability for free as part of the protocol used between
the apps & daemon.
Does RHEV-M actually use the remote libvirt interface? I assume
it'll
talk to vdsm via some protocol and vdsm will use the local libvirt API.
That is correct. That doesn't mean the remote interface isn't useful.
I suspect most uses of libvirt are actually local uses.
A key part of libvirt is interoperability between applications. So an app
like vsdm can be using libvirt locally, while at the same time virt-top
can be monitoring guests on a set of machines remotely, or virt-viewer can
be run from your desktop talking to libvirt remotely to discover the VNC
console details. For flexibility there are several options for remotely
managing VMs. The native remote RPC system. The libvirt QPid/QMF model.
The CIM binding. Application specific remote protocols. Since they all
eventually talk to the same libvirtd daemon, their use is not mutually
exclusive which is a good thing.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
9:26 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 08:58 AM, Daniel P. Berrange wrote:
On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote:
> On 04/26/2010 08:41 AM, Avi Kivity wrote:
>
>
>>> (3) The system management application can certainly create whatever
>>> context it wants to launch a vm from. It's comes down to who's
>>> responsible for creating the context the guest runs under. I think
>>> doing that at the libvirt level takes away a ton of flexibility from
>>> the management application.
>>>
>> If you want to push the flexibility slider all the way to the right
>> you get bare qemu. It exposes 100% of qemu capabilities. And it's
>> not so bad these days. But it's not something that can be remoted.
>>
> As I mentioned earlier, remoting is not a very important use-case to me.
>
NB, the "remote" protocol is no different from the "local" protocol
the unprivileged app uses to talk to the privileged daemon. The only
difference is unix sockets vs tcp sockets. We essentially get the
remote access capability for free as part of the protocol used between
the apps& daemon.
I think Avi's concern is that if you expose direct launch, then that's
not really something you can usefully exploit in a remote protocol.
For instance, if you're local, you can call setuid() before invoking the
launch API whereas if you're remote, you will always launch guests as
the context of the remoting daemon.
This makes the remote API second-class in terms of functionality.
Regards,
Anthony Liguori
> Does RHEV-M actually use the remote libvirt interface? I assume it'll
> talk to vdsm via some protocol and vdsm will use the local libvirt API.
>
That is correct. That doesn't mean the remote interface isn't useful.
> I suspect most uses of libvirt are actually local uses.
>
A key part of libvirt is interoperability between applications. So an app
like vsdm can be using libvirt locally, while at the same time virt-top
can be monitoring guests on a set of machines remotely, or virt-viewer can
be run from your desktop talking to libvirt remotely to discover the VNC
console details. For flexibility there are several options for remotely
managing VMs. The native remote RPC system. The libvirt QPid/QMF model.
The CIM binding. Application specific remote protocols. Since they all
eventually talk to the same libvirtd daemon, their use is not mutually
exclusive which is a good thing.
Daniel
9:32 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Mon, Apr 26, 2010 at 09:26:55AM -0500, Anthony Liguori wrote:
On 04/26/2010 08:58 AM, Daniel P. Berrange wrote:
>On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote:
>
>>On 04/26/2010 08:41 AM, Avi Kivity wrote:
>>
>>
>>>>(3) The system management application can certainly create whatever
>>>>context it wants to launch a vm from. It's comes down to who's
>>>>responsible for creating the context the guest runs under. I think
>>>>doing that at the libvirt level takes away a ton of flexibility from
>>>>the management application.
>>>>
>>>If you want to push the flexibility slider all the way to the right
>>>you get bare qemu. It exposes 100% of qemu capabilities. And it's
>>>not so bad these days. But it's not something that can be remoted.
>>>
>>As I mentioned earlier, remoting is not a very important use-case to me.
>>
>NB, the "remote" protocol is no different from the "local"
protocol
>the unprivileged app uses to talk to the privileged daemon. The only
>difference is unix sockets vs tcp sockets. We essentially get the
>remote access capability for free as part of the protocol used between
>the apps& daemon.
>
I think Avi's concern is that if you expose direct launch, then that's
not really something you can usefully exploit in a remote protocol.
For instance, if you're local, you can call setuid() before invoking the
launch API whereas if you're remote, you will always launch guests as
the context of the remoting daemon.
This makes the remote API second-class in terms of functionality.
It is not entirely outside the realms of possibility to consider
making a way for libvirt to attach to a QEMU spawned by a 3rd
party. eg,
virsh qemu-attach /path/to/monitor PID
Converting the QEMU config back into Libvirt XML is not trivial,
but neither is it impossible. Particularly now we have the custom
<qemu:arg> syntax for things libvirt can't represent, we can
simply use that for bits we can't reverse map to XML. There's also
the UUID/name uniqueness, but we could fail the attach operation
if that wwas violated. This would let you use most of the other
libvirt APIs remotely from any app, even if the VM was spawned by
another one.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
4:59 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Sun, Apr 25, 2010 at 08:53:17PM -0500, Anthony Liguori wrote:
On 04/25/2010 06:51 AM, Avi Kivity wrote:
> Qemu is special due to the nonexistence of qemud.
>
>Why is sVirt implemented in libvirt? it's not the logical place for
>it; rather the logical place doesn't exist.
sVirt is not just implemented in libvirt. libvirt implements a
mechanism to set the context of a given domain and dynamically label
it's resources to isolate it.
The reason it has to assign a context to a given domain is that all
domains are launched from the same security context (the libvirtd
context) as the original user's context (the consumer of the libvirt
API) has been lost via the domain socket interface.
If you used the /session URL, then the domain would have the security
context of whomever created the guest which means that dynamic labelling
of the resources wouldn't be necessary (you would just do static labelling).
That is not correct. You do *not* ever want the guests to have the same
security context as the thing that created them, because that would allow
the guest to access & compromise resources belonging to the management app.
Every guest must always have a unique context. The libvirt+sVirt dynamic
labelling with unique contexts is applied even for the /session mode.
The thing you save with the /session mode, is not having todo the user/group
ownership management; sVirt labelling is always required.
This is certainly a more secure model and it's a feature of qemu
that I
really wish didn't get lost in libvirt. Again, /session can do this too
but right now, /session really isn't usable in libvirt for qemu.
If you really want the qemu instance to inherit the context of the mgmt
app, then you can just declare in the guest XML that it should use a
static label, and pass in the apps' own label. This is *not* a more secure
model though.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
8:13 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 04:59 AM, Daniel P. Berrange wrote:
On Sun, Apr 25, 2010 at 08:53:17PM -0500, Anthony Liguori wrote:
> On 04/25/2010 06:51 AM, Avi Kivity wrote:
>
>> Qemu is special due to the nonexistence of qemud.
>>
>> Why is sVirt implemented in libvirt? it's not the logical place for
>> it; rather the logical place doesn't exist.
>>
> sVirt is not just implemented in libvirt. libvirt implements a
> mechanism to set the context of a given domain and dynamically label
> it's resources to isolate it.
>
> The reason it has to assign a context to a given domain is that all
> domains are launched from the same security context (the libvirtd
> context) as the original user's context (the consumer of the libvirt
> API) has been lost via the domain socket interface.
>
> If you used the /session URL, then the domain would have the security
> context of whomever created the guest which means that dynamic labelling
> of the resources wouldn't be necessary (you would just do static labelling).
>
That is not correct. You do *not* ever want the guests to have the same
security context as the thing that created them, because that would allow
the guest to access& compromise resources belonging to the management app.
You assume that the management app is not smart enough to create a new
context for the guest to run in.
> This is certainly a more secure model and it's a feature of
qemu that I
> really wish didn't get lost in libvirt. Again, /session can do this too
> but right now, /session really isn't usable in libvirt for qemu.
>
If you really want the qemu instance to inherit the context of the mgmt
app, then you can just declare in the guest XML that it should use a
static label, and pass in the apps' own label. This is *not* a more secure
model though.
There is more context than just selinux labelling. The problem with the
daemon model is that to create a guest, you start with a lower set of
privileges, escalate your privileges (by talking to libvirtd), then
lower privileges to launch a guest. Running a guest is essentially
running arbitrary code (since you can set the emulator path) so now
you've provided an environment where a user can launch arbitrary code as
a different user in a different security context.
There is a new attack surface here. I think it's undeniable that there
is certainly the possibility that something goes wrong and a user will
find a way to escalate it's privileges.
Compare that to a direct launch model. There is not new attack
surface. The user's privileges never increase. In fact, what's most
likely to happen is that a caller will drop some of it's privileges
before launching a guest.
Regards,
Anthony Liguori
Daniel
8:31 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Mon, Apr 26, 2010 at 08:13:03AM -0500, Anthony Liguori wrote:
On 04/26/2010 04:59 AM, Daniel P. Berrange wrote:
>On Sun, Apr 25, 2010 at 08:53:17PM -0500, Anthony Liguori wrote:
>
>>On 04/25/2010 06:51 AM, Avi Kivity wrote:
>>
>>> Qemu is special due to the nonexistence of qemud.
>>>
>>>Why is sVirt implemented in libvirt? it's not the logical place for
>>>it; rather the logical place doesn't exist.
>>>
>>sVirt is not just implemented in libvirt. libvirt implements a
>>mechanism to set the context of a given domain and dynamically label
>>it's resources to isolate it.
>>
>>The reason it has to assign a context to a given domain is that all
>>domains are launched from the same security context (the libvirtd
>>context) as the original user's context (the consumer of the libvirt
>>API) has been lost via the domain socket interface.
>>
>>If you used the /session URL, then the domain would have the security
>>context of whomever created the guest which means that dynamic labelling
>>of the resources wouldn't be necessary (you would just do static
>>labelling).
>>
>That is not correct. You do *not* ever want the guests to have the same
>security context as the thing that created them, because that would allow
>the guest to access& compromise resources belonging to the management app.
>
You assume that the management app is not smart enough to create a new
context for the guest to run in.
>>This is certainly a more secure model and it's a feature of qemu that I
>>really wish didn't get lost in libvirt. Again, /session can do this too
>>but right now, /session really isn't usable in libvirt for qemu.
>>
>If you really want the qemu instance to inherit the context of the mgmt
>app, then you can just declare in the guest XML that it should use a
>static label, and pass in the apps' own label. This is *not* a more secure
>model though.
>
There is more context than just selinux labelling. The problem with the
daemon model is that to create a guest, you start with a lower set of
privileges, escalate your privileges (by talking to libvirtd), then
lower privileges to launch a guest. Running a guest is essentially
running arbitrary code (since you can set the emulator path) so now
you've provided an environment where a user can launch arbitrary code as
a different user in a different security context.
No matter what emulator path is provided, that will never be run with
elevated privileges unless the host sysadmin has explicitly configured
that capability. The QEMU binaries always run in a dedicated account
which is completely unprivileged. This setuid is done *before* QEMU
is even exec()'d. This is one of the reasons why we don't use the
QEMU -runas argument, which would allow custom emulator binaries to
potentially gain evelated privileges.
There is a new attack surface here. I think it's undeniable that
there
is certainly the possibility that something goes wrong and a user will
find a way to escalate it's privileges.
Of course all software has bugs.
Compare that to a direct launch model. There is not new attack
surface. The user's privileges never increase. In fact, what's most
likely to happen is that a caller will drop some of it's privileges
before launching a guest.
What you describe is not inherant to the daemon model. This is why we have
two separate models in libvirt. The system instance is pre-spawned with
high privileges, to allow use of hosts resources which require high
privileges to access. The session instance is auto-spawned when the app
connects to libvirt, thus it inherits the privileges of the app that is
using it.
I don't deny that the system instance has a new attack surface, because it
is running privileged. If the app needs to connect VMs to privileged
resources, then the architecture has to have some privileged component
to give access to those resoruces. You don't want the VM to be privileged,
nor the whole management app to be privileged. The system daemon is thus the
arbitrator for this privileged access. If you don't need todo anything that
requires higher privileges though, just use the session instance which
always matches the apps privileges.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
8:43 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 08:31 AM, Daniel P. Berrange wrote:
What you describe is not inherant to the daemon model. This is why we
have
two separate models in libvirt. The system instance is pre-spawned with
high privileges, to allow use of hosts resources which require high
privileges to access. The session instance is auto-spawned when the app
connects to libvirt, thus it inherits the privileges of the app that is
using it.
I don't deny that the system instance has a new attack surface, because it
is running privileged. If the app needs to connect VMs to privileged
resources, then the architecture has to have some privileged component
to give access to those resoruces. You don't want the VM to be privileged,
nor the whole management app to be privileged. The system daemon is thus the
arbitrator for this privileged access. If you don't need todo anything that
requires higher privileges though, just use the session instance which
always matches the apps privileges.
I regret saying "more secure" because I think it's a difficult concept
to really quantify and that makes it hard to meaningfully discuss.
The reason I lean toward the direct launch model is that it gives the
user a lot of flexibility in terms of using things like namespaces, DAC,
cgroups, capabilities, etc. A lot of potential features are lost when
you do indirect launch because you have to teach the daemon how to
support each of these features.
Regards,
Anthony Liguori
Daniel
9:01 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 04:43 PM, Anthony Liguori wrote:
The reason I lean toward the direct launch model is that it gives the
user a lot of flexibility in terms of using things like namespaces,
DAC, cgroups, capabilities, etc. A lot of potential features are lost
when you do indirect launch because you have to teach the daemon how
to support each of these features.
But what's the alternative? Teach the user how to do all these things?
It's infinitely flexible, but it's not an API you can give to a
management tool developer.
--
error compiling committee.c: too many arguments to function
9:19 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 09:01 AM, Avi Kivity wrote:
On 04/26/2010 04:43 PM, Anthony Liguori wrote:
> The reason I lean toward the direct launch model is that it gives the
> user a lot of flexibility in terms of using things like namespaces,
> DAC, cgroups, capabilities, etc. A lot of potential features are
> lost when you do indirect launch because you have to teach the daemon
> how to support each of these features.
But what's the alternative? Teach the user how to do all these things?
You can expose layers of API. The lowest layer makes no changes to the
security context. A higher (optional) layer could do dynamic labelling.
It's infinitely flexible, but it's not an API you can give to
a
management tool developer.
I think the goal of a management API should be to make common things
very simple to do but not preclude doing even the most advanced things.
Regards,
Anthony Liguori
9:25 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 05:19 PM, Anthony Liguori wrote:
On 04/26/2010 09:01 AM, Avi Kivity wrote:
> On 04/26/2010 04:43 PM, Anthony Liguori wrote:
>> The reason I lean toward the direct launch model is that it gives
>> the user a lot of flexibility in terms of using things like
>> namespaces, DAC, cgroups, capabilities, etc. A lot of potential
>> features are lost when you do indirect launch because you have to
>> teach the daemon how to support each of these features.
>
> But what's the alternative? Teach the user how to do all these things?
You can expose layers of API. The lowest layer makes no changes to
the security context. A higher (optional) layer could do dynamic
labelling.
Or a library that the user-written launcher calls. Or a plugin that
qemud calls.
> It's infinitely flexible, but it's not an API you can
give to a
> management tool developer.
I think the goal of a management API should be to make common things
very simple to do but not preclude doing even the most advanced things.
Agreed.
--
error compiling committee.c: too many arguments to function
9:28 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 09:25 AM, Avi Kivity wrote:
On 04/26/2010 05:19 PM, Anthony Liguori wrote:
> On 04/26/2010 09:01 AM, Avi Kivity wrote:
>> On 04/26/2010 04:43 PM, Anthony Liguori wrote:
>>> The reason I lean toward the direct launch model is that it gives
>>> the user a lot of flexibility in terms of using things like
>>> namespaces, DAC, cgroups, capabilities, etc. A lot of potential
>>> features are lost when you do indirect launch because you have to
>>> teach the daemon how to support each of these features.
>>
>> But what's the alternative? Teach the user how to do all these things?
>
> You can expose layers of API. The lowest layer makes no changes to
> the security context. A higher (optional) layer could do dynamic
> labelling.
Or a library that the user-written launcher calls. Or a plugin that
qemud calls.
A plugin would lose the security context. It could attempt to recreate
it that seems like a lot of unnecessary complexity.
Regards,
Anthony Liguori
9:38 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 05:28 PM, Anthony Liguori wrote:
> Or a library that the user-written launcher calls. Or a plugin
that
> qemud calls.
A plugin would lose the security context. It could attempt to
recreate it that seems like a lot of unnecessary complexity.
A plugin would create the security context instead of the launcher.
Currently security contexts are created by the login process. We could
easily reuse that. Any other security context code would be custom
written; so it can be written as a qemud plugin instead of a bit of code
that goes before a qemu launch.
--
error compiling committee.c: too many arguments to function
9:48 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 09:38 AM, Avi Kivity wrote:
On 04/26/2010 05:28 PM, Anthony Liguori wrote:
>> Or a library that the user-written launcher calls. Or a plugin that
>> qemud calls.
>
>
> A plugin would lose the security context. It could attempt to
> recreate it that seems like a lot of unnecessary complexity.
>
A plugin would create the security context instead of the launcher.
Currently security contexts are created by the login process.
It's not always that centralized. An initial context is created by the
login process, but then later something may come along and create a
network namespace as part of containerization.
We could easily reuse that. Any other security context code would
be custom written; so it can be written as a qemud plugin instead of a
bit of code that goes before a qemu launch.
I think we're mostly in agreement with respect to the need to have more
control over the security context the qemu runs in. Whether it's
launched via a daemon or directly I think is an implementation detail
that we can debate when we get closer to an actual implementation.
Regards,
Anthony Liguori
9:51 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/26/2010 05:48 PM, Anthony Liguori wrote:
> We could easily reuse that. Any other security context code would
> be custom written; so it can be written as a qemud plugin instead of
> a bit of code that goes before a qemu launch.
I think we're mostly in agreement with respect to the need to have
more control over the security context the qemu runs in. Whether it's
launched via a daemon or directly I think is an implementation detail
that we can debate when we get closer to an actual implementation.
Good, as I haven't decided yet which side I'm on.
--
error compiling committee.c: too many arguments to function
9:34 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Fri, Apr 23, 2010 at 08:48:51AM -0500, Anthony Liguori wrote:
On 04/23/2010 07:48 AM, Avi Kivity wrote:
>On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>>real API. Say, adding a device libvirt doesn't know about or
>>>stopping the VM
>>>while libvirt thinks it's still running or anything like that.
>> Another problem is issuing Monitor commands that could confuse
>>libvirt's
>>
>>We need to make libvirt and qemu smarter.
>>
>>We already face this problem today with multiple libvirt users. This
>>is why sophisticated management mechanisms (like LDAP) have
>>mechanisms to do transactions or at least a series of atomic operations.
>
>And people said qmp/json was overengineered...
>
>But seriously, transactions won't help anything. qemu maintains
>state, and when you have two updaters touching a shared variable not
>excepting each other to, things break, no matter how much locking
>there is.
Let's consider some concrete examples. I'm using libvirt and QMP and in
QMP, I want to hot unplug a device.
Today, I do this by listing the pci devices, and issuing a pci_del that
takes a PCI address. This is intrinsically racy though because in the
worst case scenario, in between when I enumerate pci devices and do the
pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add
within libvirt of a completely different device.
This is what already happens with any QEMU >= 0.12, where libvirt uses
the new -device syntax with its 'id' parameter for all devices, and
then uses 'device_id $ID' for unplug. The app still has to be careful
it doesn't try to add a device using the same naming scheme as a
device libvirt uses. This is easy enough though if they prepend some
random string to all device IDs thy use, that won't clash with libvirt
device ID names.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
10:43 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
Anthony Liguori <anthony(a)codemonkey.ws> writes:
On 04/23/2010 07:48 AM, Avi Kivity wrote:
> On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>> real API. Say, adding a device libvirt doesn't know about or
>>> stopping the VM
>>> while libvirt thinks it's still running or anything like that.
>> Another problem is issuing Monitor commands that could confuse
>> libvirt's
>>
>> We need to make libvirt and qemu smarter.
>>
>> We already face this problem today with multiple libvirt users.
>> This is why sophisticated management mechanisms (like LDAP) have
>> mechanisms to do transactions or at least a series of atomic
>> operations.
>
> And people said qmp/json was overengineered...
>
> But seriously, transactions won't help anything. qemu maintains
> state, and when you have two updaters touching a shared variable not
> excepting each other to, things break, no matter how much locking
> there is.
Let's consider some concrete examples. I'm using libvirt and QMP and
in QMP, I want to hot unplug a device.
Today, I do this by listing the pci devices, and issuing a pci_del
that takes a PCI address. This is intrinsically racy though because
in the worst case scenario, in between when I enumerate pci devices
and do the pci_del in QMP, in libvirt, I've done a pci_del and then a
pci_add within libvirt of a completely different device.
There are a few ways to solve this, the simplest being that we give
devices unique ids that are never reused and instead of pci_del taking
a pci bus address, it takes a device id. That would address this
race.
For what it's worth, that's how device_del works, except the ID is
chosen by the user, who is free to reuse them.
Really tangential to this thread, but it's worth repeating anyway: use
device_add & friends, not pci_add & friends.
[...]
1:45 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:
On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
> Hello,
> In response to a lot of the talk of qemud lately on qemu-devel, the
> libvirt community would like to put forward a proposal to help enable
> debug/advanced options when using various hypervisors. The goals of
> this API are:
>
> 1) To enable more rapid access to hypervisor features before proper
> libvirt API's are designed around them.
> 2) To facilitate debugging and access to advanced features that may
> not fit into the normal libvirt world-view.
>
> Caveats:
> 1) Unlike other libvirt API's, this one will explicitly *not* be
> guaranteed ABI/API compatible between libvirt updates.
>
I think we'd still aim to keep the public API stable. The bit that we
can't guarentee is the interactions with QEMU& the libvirt driver.
eg, if someone was using the API to send text monitor commands to
QEMU, and the next libvirt release switched to JSON mode, that use
would break. The API would still be valid, but the way they use it
might not be. Similarly if they add some custom extra command line
argument, this could potentially conflict with an extra command line
arg a subsquent libvirt release used. eg, they used -device to add
a PCI card with a specific PCI address. Then next libvirt release
specifies this same PCI address and then you get a clash.
So in both cases the API and XML format can be reasonably guarenteed
between releases. What we can't guarentee is that usage of these
features will be reliable across releases of libvirt.
> 2) Again unlike other libvirt API's, access and configuration of
> the debug section of a domain will be highly hypervisor dependent.
> 3) Application developers will be strongly discouraged from using
> this API because of the above 2 issues. To help in this, the
> API's will be in a separate library that developers will explicitly
> have to link to, and it will have a different (but largely compatible)
> wire protocol.
>
In terms of wire protocol, this would still have to run over the same
existing data channel, because we don't want apps to have to open up
multiple connections. Fortunately our protocol is designed to allow
this kind of extension
struct remote_message_header {
unsigned prog; /* REMOTE_PROGRAM */
unsigned vers; /* REMOTE_PROTOCOL_VERSION */
remote_procedure proc; /* REMOTE_PROC_x */
remote_message_type type;
unsigned serial; /* Serial number of message. */
remote_message_status status;
};
We currently have a single program 'REMOTE_PROGRAM' and its associated
'vers' and 'proc' numbers. What we would be doing is to define a new
program, say QEMU_PROGRAM. The 'remote_procedure proc' numbers can thus
start again from '1' without clashing with any of the existing APIs.
All the RPC marshalling/demarshalling code should work more or less
unchanged. We just hook in a different function dispatch table in the
daemon for that program.
> 4) We don't expect this API to solve all of the issues brought up
> during the qemud discussion. Our initial goal is just to give
> ready access of the qemu command-line and monitor to developers.
>
> With that being said, our initial proposal follows. We expect this
> to evolve over time as we get more feedback, but we think this
> proposal addresses at least 2 of the major pain points qemu developers
> have while trying to use libvirt.
>
> The initial debug XML for qemu would be:
>
I wouldn't call them 'debug' options, since that's just one of possible
use cases for this. What we're really doing there is exposing hypervisor
specific features, so we should just be upfront about that in our
description / naming.
> <domain type='kvm'>
> <name>myguest</name>
> ...
> <debug>
> <monitorpassthrough/>
> <commandline>
> <extra>qemu arguments</extra>
> <alter option="optname">
> <rename>newname</rename>
> <match>REGEXP</match>
> <modify>foo=on</modify>
> <extra>-bar</extra>
> </alter>
> </commandline>
> </debug>
> </domain>
>
The concept of command line& monitor is something that is QEMU specific
and thus is not suitable for the primary XML schema. IMHO, this needs to be
done as a separate schema, linked in via an XML namespace. For example
<domain type='kvm'
xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
<name>myguest</name>
...
<qemu:commandline>
<qemu:arg>-device</arg>
<qemu:arg>lsi</arg>
</qemu:commandline>
</domain>
I think it's problematic to focus too much on command line arguments.
We are not introducing new command line arguments to qemu for the most
part that aren't usable in the config file.
What I really think we would like, is a way to express the qemu
configuration file in terms of XML and then to use snippets of that
within an XML namespace. For instance:
[cpudef]
name = "Opteron_G3"
level = "5"
vendor = "AuthenticAMD"
...
Could convert to:
<cpudef>
<name>Operon_G3</name>
<level>5</level>
<vendor>AuthenticAMD</vendor>
...
</cpudef>
Which in turn, could be embedded as:
<domain type='kvm' xmlns:qemu='http://qemu.org/schemas/qemu/1.0'>
<name>myguest</name>
...
<qemu:cpudef>
<name>Operon_G3</name>
<level>5</level>
<vendor>AuthenticAMD</vendor>
</qemu:cpudef>
</domain>
With respect to injecting QMP commands directly, I think the proposed
debug API is probably reasonable. We could build a libqemu that used
that API as a transport which means that one could use libqemu and
libvirt simultaneously which is certainly a key requirement of mine.
I think it's important that it's a dedicated monitor session though. It
shouldn't just be injecting commands within an existing QMP session IMHO.
Regards,
Anthony Liguori
Regards,
Anthony Liguori
2:10 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/22/2010 01:45 PM, Anthony Liguori wrote:
Which in turn, could be embedded as:
<domain type='kvm' xmlns:qemu='http://qemu.org/schemas/qemu/1.0'>
<name>myguest</name>
...
<qemu:cpudef>
<name>Operon_G3</name>
<level>5</level>
<vendor>AuthenticAMD</vendor>
</qemu:cpudef>
</domain>
With respect to injecting QMP commands directly, I think the proposed
debug API is probably reasonable. We could build a libqemu that used
that API as a transport which means that one could use libqemu and
libvirt simultaneously which is certainly a key requirement of mine.
I think it's important that it's a dedicated monitor session though.
It shouldn't just be injecting commands within an existing QMP session
IMHO.
Another important thing to mention is that this type of conversion has
two purposes. A user can inject qemu config snippets into the libvirt
description, but libvirt can also inject qemu config snippets that it
cannot represent via it's own XML for the user.
For instance, let's say that a user adds a disk via QMP with an option
that libvirt doesn't recognize. When a virsh dumpxml is executed,
libvirt could display the XML using the <qemu:drive> tag instead of the
<disk> tag that it normally would use.
Regards,
Anthony Liguori
Regards,
Anthony Liguori
5:28 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote:
On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:
>On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
>
>
>><domain type='kvm'>
>> <name>myguest</name>
>> ...
>> <debug>
>> <monitorpassthrough/>
>> <commandline>
>> <extra>qemu arguments</extra>
>> <alter option="optname">
>> <rename>newname</rename>
>> <match>REGEXP</match>
>> <modify>foo=on</modify>
>> <extra>-bar</extra>
>> </alter>
>> </commandline>
>> </debug>
>></domain>
>>
>The concept of command line& monitor is something that is QEMU specific
>and thus is not suitable for the primary XML schema. IMHO, this needs to be
>done as a separate schema, linked in via an XML namespace. For example
>
> <domain type='kvm'
> xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
> <name>myguest</name>
> ...
> <qemu:commandline>
> <qemu:arg>-device</arg>
> <qemu:arg>lsi</arg>
> </qemu:commandline>
> </domain>
>
I think it's problematic to focus too much on command line arguments.
We are not introducing new command line arguments to qemu for the most
part that aren't usable in the config file.
Currently libvirt doesn't use any config file at all, so command line
is the only possible option. We're going to evaluate switching to the
config file at some point & so can easily add further XML options to
allow direct setting of config file entries at that point. There's no
problem supporting both command line & config file syntax in the XML.
The command line does have the immediate advantage that 99% of all
documentation about QEMU on the web today refers to command line args
so that's what people are most familiar with.
With respect to injecting QMP commands directly, I think the proposed
debug API is probably reasonable. We could build a libqemu that used
that API as a transport which means that one could use libqemu and
libvirt simultaneously which is certainly a key requirement of mine.
I think it's important that it's a dedicated monitor session though. It
shouldn't just be injecting commands within an existing QMP session IMHO.
I think the opposite actually. If libvirt had two open monitor connections,
one for normal use & one for injection, then its open to racey usage where
2 monitor commands may be issued concurrently & it is tricky to determine
just which will be processed first, with all the scope for unexpected
behaviour this entails. With a single monitor connection, libvirt's current
locking model for the monitor ensures that QMP monitor commands are reliably
serialized onto the wire, giving unambiguous behaviour.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
8:40 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/23/2010 05:28 AM, Daniel P. Berrange wrote:
On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote:
> On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:
>
>> On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
>>
>>
>>
>>> <domain type='kvm'>
>>> <name>myguest</name>
>>> ...
>>> <debug>
>>> <monitorpassthrough/>
>>> <commandline>
>>> <extra>qemu arguments</extra>
>>> <alter option="optname">
>>> <rename>newname</rename>
>>> <match>REGEXP</match>
>>> <modify>foo=on</modify>
>>> <extra>-bar</extra>
>>> </alter>
>>> </commandline>
>>> </debug>
>>> </domain>
>>>
>>>
>> The concept of command line& monitor is something that is QEMU specific
>> and thus is not suitable for the primary XML schema. IMHO, this needs to be
>> done as a separate schema, linked in via an XML namespace. For example
>>
>> <domain type='kvm'
>> xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
>> <name>myguest</name>
>> ...
>> <qemu:commandline>
>> <qemu:arg>-device</arg>
>> <qemu:arg>lsi</arg>
>> </qemu:commandline>
>> </domain>
>>
>>
> I think it's problematic to focus too much on command line arguments.
> We are not introducing new command line arguments to qemu for the most
> part that aren't usable in the config file.
>
Currently libvirt doesn't use any config file at all, so command line
is the only possible option.
The config file is really just a structured command line argument so the
two are really equivalent.
From an XML perspective, the advantage of using the config file format
is that it's very well structured whereas the command line arguments
aren't. It's the difference between:
<qemu:commandline>
<arg>-drive</arg>
<arg>file=foo.img,if=virtio,cache=off</arg>
</qemu:commandline>
And:
<qemu:drive>
<file>foo.img</file>
<if>virtio</if>
<cache>off</cache>
</qemu:drive>
The later being much more friendly for things like XPath and XQuery.
We're going to evaluate switching to the
config file at some point& so can easily add further XML options to
allow direct setting of config file entries at that point.
I don't necessary think there's a super compelling reason for you to
switch to pure config. Some things you'll have to do via -device and
-set but it's still reasonable to keep using the command line. A key
requirement of the config file effort was that everything that can be
done via config file be doable via command line.
> With respect to injecting QMP commands directly, I think the
proposed
> debug API is probably reasonable. We could build a libqemu that used
> that API as a transport which means that one could use libqemu and
> libvirt simultaneously which is certainly a key requirement of mine.
>
> I think it's important that it's a dedicated monitor session though. It
> shouldn't just be injecting commands within an existing QMP session IMHO.
>
I think the opposite actually. If libvirt had two open monitor connections,
one for normal use& one for injection, then its open to racey usage where
2 monitor commands may be issued concurrently& it is tricky to determine
just which will be processed first, with all the scope for unexpected
behaviour this entails.
Can you give an explicit example? The nature of this debug extension is
such that I don't think 1 monitor really helps this problem...
With a single monitor connection, libvirt's current
locking model for the monitor ensures that QMP monitor commands are reliably
serialized onto the wire, giving unambiguous behaviour.
Except that libvirt doesn't know what the side effects of the debug
commands are so it's intrinsically ambiguous :-)
Regards,
Anthony Liguori
Regards,
Daniel
9:21 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On Fri, Apr 23, 2010 at 08:40:49AM -0500, Anthony Liguori wrote:
On 04/23/2010 05:28 AM, Daniel P. Berrange wrote:
>On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote:
>
>>On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:
>>
>>>On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
>>>
>>>
>>>
>>>><domain type='kvm'>
>>>> <name>myguest</name>
>>>> ...
>>>> <debug>
>>>> <monitorpassthrough/>
>>>> <commandline>
>>>> <extra>qemu arguments</extra>
>>>> <alter option="optname">
>>>> <rename>newname</rename>
>>>> <match>REGEXP</match>
>>>> <modify>foo=on</modify>
>>>> <extra>-bar</extra>
>>>> </alter>
>>>> </commandline>
>>>> </debug>
>>>></domain>
>>>>
>>>>
>>>The concept of command line& monitor is something that is QEMU
specific
>>>and thus is not suitable for the primary XML schema. IMHO, this needs to
>>>be
>>>done as a separate schema, linked in via an XML namespace. For example
>>>
>>> <domain type='kvm'
>>> xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
>>> <name>myguest</name>
>>> ...
>>> <qemu:commandline>
>>> <qemu:arg>-device</arg>
>>> <qemu:arg>lsi</arg>
>>> </qemu:commandline>
>>> </domain>
>>>
>>>
>>I think it's problematic to focus too much on command line arguments.
>>We are not introducing new command line arguments to qemu for the most
>>part that aren't usable in the config file.
>>
>Currently libvirt doesn't use any config file at all, so command line
>is the only possible option.
The config file is really just a structured command line argument so the
two are really equivalent.
From an XML perspective, the advantage of using the config file format
is that it's very well structured whereas the command line arguments
aren't. It's the difference between:
<qemu:commandline>
<arg>-drive</arg>
<arg>file=foo.img,if=virtio,cache=off</arg>
</qemu:commandline>
And:
<qemu:drive>
<file>foo.img</file>
<if>virtio</if>
<cache>off</cache>
</qemu:drive>
The later being much more friendly for things like XPath and XQuery.
> We're going to evaluate switching to the
>config file at some point& so can easily add further XML options to
>allow direct setting of config file entries at that point.
I don't necessary think there's a super compelling reason for you to
switch to pure config. Some things you'll have to do via -device and
-set but it's still reasonable to keep using the command line. A key
requirement of the config file effort was that everything that can be
done via config file be doable via command line.
The main reason we're interested in the config is that it uses a better
structured format, so we don't have such hairy escaping problems to deal
with. Also, the command line is getting outrageously long these days
with -device syntax.
>>With respect to injecting QMP commands directly, I think the
proposed
>>debug API is probably reasonable. We could build a libqemu that used
>>that API as a transport which means that one could use libqemu and
>>libvirt simultaneously which is certainly a key requirement of mine.
>>
>>I think it's important that it's a dedicated monitor session though. It
>>shouldn't just be injecting commands within an existing QMP session IMHO.
>>
>I think the opposite actually. If libvirt had two open monitor connections,
>one for normal use& one for injection, then its open to racey usage where
>2 monitor commands may be issued concurrently& it is tricky to determine
>just which will be processed first, with all the scope for unexpected
>behaviour this entails.
Can you give an explicit example? The nature of this debug extension is
such that I don't think 1 monitor really helps this problem...
Say libvirt is running a 'offline core dump' operation. This consists of
us invoking
stop
migrate exec:cat > foo.dump
cont
I don't want other debug commands accidentally being issued in between
these steps. These 3 commands are in essence considered transactional
block. Internally our locking model ensures that no other APIs can be
invoked in this monitor connection during this sequence. Thus if someone
uses the libvirt debug API to issue a monitor command, thus command will
be blocked until the last 'cont' command is issued here. This provides us
a known command ordering in the logs.
> With a single monitor connection, libvirt's current
>locking model for the monitor ensures that QMP monitor commands are
>reliably
>serialized onto the wire, giving unambiguous behaviour.
Except that libvirt doesn't know what the side effects of the debug
commands are so it's intrinsically ambiguous :-)
This is a different ambiguity, about the semantic results of the commands,
where as I'm refering to the execution order. If I look at a libvirt log
file and see a set of JSON commands logged, I want to know that this ordering
from the logs, was indeed the same as order in which qemu processed them. If
you have two separate monitor connection you can't be sure of the order of
execution. It is key for our bug troubleshooting that given a libvirt log
file, we can replay the JSON commands again and get the same results. Two
monitor connections is just increasing complexity of code without any
tangible benefit.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
1:33 p.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/23/2010 09:21 AM, Daniel P. Berrange wrote:
Say libvirt is running a 'offline core dump' operation. This consists of
us invoking
stop
migrate exec:cat> foo.dump
cont
I don't want other debug commands accidentally being issued in between
these steps. These 3 commands are in essence considered transactional
block. Internally our locking model ensures that no other APIs can be
invoked in this monitor connection during this sequence. Thus if someone
uses the libvirt debug API to issue a monitor command, thus command will
be blocked until the last 'cont' command is issued here. This provides us
a known command ordering in the logs.
Ah, so what'd I'd suggest is that you still use two monitors but that
you use the same locking for both. The reason a second monitor is
useful is that you can allow different event masks to be setup on the
second monitor which I think a QMP client will expect.
I don't see any trouble with respect to command ordering here. libvirt
is ultimately the one sending commands to monitor. You know which one
was sent and which one completed in what order.
>> With a single monitor connection, libvirt's current
>> locking model for the monitor ensures that QMP monitor commands are
>> reliably
>> serialized onto the wire, giving unambiguous behaviour.
>>
> Except that libvirt doesn't know what the side effects of the debug
> commands are so it's intrinsically ambiguous :-)
>
This is a different ambiguity, about the semantic results of the commands,
where as I'm refering to the execution order. If I look at a libvirt log
file and see a set of JSON commands logged, I want to know that this ordering
from the logs, was indeed the same as order in which qemu processed them. If
you have two separate monitor connection you can't be sure of the order of
execution. It is key for our bug troubleshooting that given a libvirt log
file, we can replay the JSON commands again and get the same results. Two
monitor connections is just increasing complexity of code without any
tangible benefit.
I think you're assuming direct access to the second monitor? I'm not
suggesting that. I'm suggesting that libvirt is still the one
submitting commands to the second monitor and that it submits those
commands in lock step.
Regards,
Anthony Liguori
9:50 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/23/2010 09:33 PM, Anthony Liguori wrote:
> This is a different ambiguity, about the semantic results of the
> commands,
> where as I'm refering to the execution order. If I look at a libvirt log
> file and see a set of JSON commands logged, I want to know that this
> ordering
> from the logs, was indeed the same as order in which qemu processed
> them. If
> you have two separate monitor connection you can't be sure of the
> order of
> execution. It is key for our bug troubleshooting that given a libvirt
> log
> file, we can replay the JSON commands again and get the same results.
> Two
> monitor connections is just increasing complexity of code without any
> tangible benefit.
I think you're assuming direct access to the second monitor? I'm not
suggesting that. I'm suggesting that libvirt is still the one
submitting commands to the second monitor and that it submits those
commands in lock step.
What about protocol extensions? For instance, pretend libvirt doesn't
support async messages, what would it do when it receives one from the
user's monitor?
--
error compiling committee.c: too many arguments to function
8:14 a.m.
New subject: [libvirt] [Qemu-devel] Re: Libvirt debug API
On 04/25/2010 09:50 AM, Avi Kivity wrote:
On 04/23/2010 09:33 PM, Anthony Liguori wrote:
>> This is a different ambiguity, about the semantic results of the
>> commands,
>> where as I'm refering to the execution order. If I look at a libvirt
>> log
>> file and see a set of JSON commands logged, I want to know that this
>> ordering
>> from the logs, was indeed the same as order in which qemu processed
>> them. If
>> you have two separate monitor connection you can't be sure of the
>> order of
>> execution. It is key for our bug troubleshooting that given a
>> libvirt log
>> file, we can replay the JSON commands again and get the same
>> results. Two
>> monitor connections is just increasing complexity of code without any
>> tangible benefit.
>
> I think you're assuming direct access to the second monitor? I'm not
> suggesting that. I'm suggesting that libvirt is still the one
> submitting commands to the second monitor and that it submits those
> commands in lock step.
>
What about protocol extensions? For instance, pretend libvirt doesn't
support async messages, what would it do when it receives one from the
user's monitor?
Protocol extensions could not be supported in this model. I agree,
that's unfortunate.
Regards,
Anthony Liguori
3:07 p.m.
On 04/09/2010 07:41 AM, Chris Lalancette wrote:
Caveats:
3) Application developers will be strongly discouraged from using
this API because of the above 2 issues. To help in this, the
API's will be in a separate library that developers will explicitly
have to link to, and it will have a different (but largely compatible)
wire protocol.
Providing two side-by-side libraries under the libvirt package umbrella
makes sense to me, but it means we need to start thinking more seriously
about library version numbers. See
https://www.redhat.com/archives/libvir-list/2010-April/msg00226.html for
some food for thought; in particular, the idea that since we create our
library (or libraries, after this proposal is enacted) via libtool, we
should be using the libtool versioning scheme for the .so files, which
would be independent from the libvirt package number. It is entirely
feasible to have a release that increments the version number of one but
not both .so, or which increments them in different manners (libvirt
increasing current and age, because it added new APIs but is still
backwards compatible, while the helper library resets age to 0 because
it added a backwards incompatible change).
--
Eric Blake eblake(a)redhat.com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
5324
days inactive
5341
days old
66 comments
13 participants
participants (13)
-
Anthony Liguori -
Avi Kivity -
Chris Lalancette -
Daniel P. Berrange -
Daniel Veillard -
Eric Blake -
Jamie Lokier -
Jes Sorensen -
Luiz Capitulino -
Markus Armbruster -
Matthias Bolte -
Paolo Bonzini -
Richard W.M. Jones