On 04/09/2010 10:27 AM, Daniel P. Berrange wrote:

The concept of command line & monitor is something that is QEMU specific and thus is not suitable for the primary XML schema. IMHO, this needs to be done as a separate schema, linked in via an XML namespace. For example

<domain type='kvm' xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0"> <name>myguest</name> ... <qemu:commandline> <qemu:arg>-device</arg> <qemu:arg>lsi</arg> </qemu:commandline> </domain>

This is a very good idea. I didn't really know about namespaces, but now that you've pointed them out I think this is a great way to partition it up. I'll start looking at this.

...

Raw access to the qemu monitor will be disabled by default; the <monitorpassthrough/> tag enables the ability to send QMP (or text, if you are using older qemu) messages straight through to the monitor. To do this there will be an additional API entry point named virDomainDebugCommand() which takes an arbitrary string and passes it to the monitor, and returns an arbitrary string as a result. Thus you could pass in either "info cpus" if using the text monitor or '{ "execute": "query-cpus" }' if using QMP.

Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with other hypervisors have different approaches for low level extension/ debug. For example, Xen would involve XenStore access, or XenD XMLRPC, etc. So this should really live in a separate API namespace which is specific to a hypervisor. For example, as a header file

#include <libvirt/libvirt-qemu.h>

Containing APIs like

int virDomainQEMUInvokeMonitor(virDomainPtr dom, const char *command, char **reply);

typedef virConnectQEMUDomainEventCallback(virConnectPtr conn, virDomainPtr dom, const char *eventname, const char *data, void *opaque) int virConnectQEMUDomainEventRegister(virConnectPtr conn, virDomainPtr dom, const char *eventname, virDomainQEMUMonitorCallback cb, void *opaque);

For an add-on library

libvirt-qemu.so

I don't think there's much to be gained from having an XML element to turn on/off use of these APIs. If an app doesn't want to use them, it can simply not link to libvirt-qemu.so

The reason I wanted to do this was mostly for debug/support reasons. That is, with this element in place we can easily tell from the dumpxml output whether a person was using the "unreliable" API's, and thus we can tell them to try and reproduce without that in place.

...

The <commandline><extra> tag does exactly what you might expect; appends the exact string to the qemu command-line.

Allowing many args at once in the <extra> blob means that libvirt will need to parse & split this up into individual args which than then be safely passed to 'exec'. It is better to specify one arg per element to avoid this fragile parsing problem.

That is true. However, it's nicer to the user (and more like the qemu command-line) to specify the options as one big string. In any case we already have the code for parsing a qemu command-line (for the native-to-xml API), so we can just adapt that.

...

The <alter> tag gets more interesting. The idea is that <alter> would allow you to modify the libvirt-generated qemu command-line in arbitrary ways. How this would work is probably best explained with some examples:

<commandline> <alter option="-net"> <rename>-netdev</rename> </alter> </commandline>

In this example, all options named -net on the qemu command-line are renamed to -netdev.

In this example, if (and only if) a -net option is seen, then -usbtablet is appended to the qemu command-line.

<commandline> <alter option="-net"> <match>\(.*name=hostnet0.*\)</match> <modify>\1,tap</modify> </alter> </commandline>

This gets more complicated (but also more powerful). In this case, any -net

I think this alteration of existing args is faaaar too complex & fragile, and way overkill. If the arg that libvirt generates isn't what someone needs, then remove the bit of the guest config responsible for that and add a complete extra arg, rather than munging the existing one.

It is complex, I agree. I also think it is pretty powerful. But if the need arises, we can always add it later. For now we'll go with just the monitor passthrough and simple command-line addition, and we can evolve from there. -- Chris Lalancette

On 04/12/2010 08:41 AM, Daniel P. Berrange wrote:

...

...

I don't think there's much to be gained from having an XML element to turn on/off use of these APIs. If an app doesn't want to use them, it can simply not link to libvirt-qemu.so

The reason I wanted to do this was mostly for debug/support reasons. That is, with this element in place we can easily tell from the dumpxml output whether a person was using the "unreliable" API's, and thus we can tell them to try and reproduce without that in place.

That doesn't tell you whether they have actually used any API or not. It is also inconvenient if you start a guest without it, and only later realize you want to use the extra APIs. If we want to track the actual usage, then the first time a direct monitor command is issued, we should simply log a warning message.

The problem with logging a message is that it is easy to lose it. What I'm trying to avoid here is debugging somebody's setup for hours only to find out that they did a "pci_del" behind libvirt's back. Maybe we can just make the <monitorpassthrough/> a read-only flag; it is ignored in the parsing, but it is set by "GetXMLDesc" when it detects that the virDomainQemuInvokeMonitor has been called. -- Chris Lalancette

Daniel P. Berrange wrote:

I think this alteration of existing args is faaaar too complex & fragile, and way overkill.

Would it not be simpler, for the target audience, for the config to contain a one-line shell script to transform particular matched arguments in any way that's wanted?

If the arg that libvirt generates isn't what someone needs, then remove the bit of the guest config responsible for that and add a complete extra arg, rather than munging the existing one.

Problem with that: You might want libvirt to think it has a device that it's controlling, still show in the GUI, still tie up with the rest of the stack, still do the management setup, but you just want to change some qemu-specific parameters. For example, cache=somethingelse for block devices, iptables=tobedefined for net devices, usbimplementation=hack for USB devices. -- Jamie

On 04/09/2010 11:30 PM, Eric Blake wrote:

Yeah, having the ability to specify an optional wrapper script, that receives the name of the normal interpreter and all the arguments the normal interpreter would have been given, sounds like the ultimate in flexibility at a minimum of xml.

You can also just have it piped into sed. Paolo

Richard W.M. Jones wrote:

On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote:

...

Daniel P. Berrange wrote:

...

I think this alteration of existing args is faaaar too complex & fragile, and way overkill.

Would it not be simpler, for the target audience, for the config to contain a one-line shell script to transform particular matched arguments in any way that's wanted?

I might be missing something but I thought you could already do this in current libvirt. ie. Change the <emulator> element [1] to point to your shell script:

<emulator>/usr/local/bin/my-qemu-wrapper.sh</emulator>

The wrapper should get called with the command line arguments and you can alter them however you like.

Isn't that right, Dan?

That forces the shell script to be relatively complicated and recognise all qemu arguments (including version specific ones) to skip the ones with parameters - and to know where to look for the real target executable is. It's not a very friendly way to add or change parts of specific subsystems, for example modifying a blockdev parameter, but it can be done that way if necessary. It's not that hard to write this for trivial extra options: <emulator>/bin/sh -c 'qemu "$0" "$@" -extra-flag'</emulator> (if that works). But for things like modifications to particular devices, wouldn't it be better to associate "add cache=special option" to a section about one of the blockdevs which libvirt knows about, rather than the awful prospect of parsing libvirt's output to guess which of three -device arguments corresponds with the second SCSI drive mentioned in the libvirt config - which might not necessarily be the second on the command line, even. Parsing libvirt output and having to guess which option corresponds to what from the libvirt config sounds very fragile and also a rather large amount of effort for something which should be easy. And then there's modifying how network tap devices are set up and other networking, outside qemu, but requested by libvirt. Doing that in <emulator> sounds like a bad idea, but script hooks are the only sane way to make networking fit every environment and requirement. Ideally without forcing the hook writer to reimplement the normal network setup from scratch just to tweak it a bit. I apologise if libvirt already provides such hooks - I haven't looked at that part of it. -- Jamie

Daniel Veillard wrote:

On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:

...

It's not that hard to write this for trivial extra options:

<emulator>/bin/sh -c 'qemu "$0" "$@" -extra-flag'</emulator>

(if that works).

That won't work because we expect the emulator to be a path to the emulator program, so yes that has to be done in the wrapper script.

Ok. Being able to pass arguments would be about infinitely more useful, so that you don't need a separate wrapper script for every individual guest configuration - you can pass options to the single wrapper for each variation. Even <emulator>/path/to/wrapper qemu-0.11</emulator> is a big improvement over <emulator>/path/to/wrapper-guest-config-6-with-qemu-0.11</emulator> ;-) Some simple but versatile hook ideas: - <emulator-append-option> (no space splitting, one option, appended) - <emulator-append-options> (space splitting multiple options) - <emulator-prepend-option> - <emulator-prepend-options> - <emulator-setenv name="NAME">VALUE</emulator-setenv> - <emulator-wrap>/path/to/script</emulator-wrap> This last one is to call the script, but pass all the options including the path to whichever version of qemu/kvm would get selected by libvirt. (I presume it has some selection mechanism, given the variety of guests which break with each version of qemu and/or qemu-kvm and/or kvm?) (It is most unfortunate that XML forces those long prefixes everywhere due to lack of contextual naming, unlike less verbose config formats :-/)

...

Parsing libvirt output and having to guess which option corresponds to what from the libvirt config sounds very fragile and also a rather large amount of effort for something which should be easy.

it's not that easy because we change the way we build the command line for qemu as qemu evolves. for example before and after the support for -device there have been a lot of changes.

I agree, which is why it's much less fragile if libvirt provides the individual parts to hooks/wrappers, if that's asked, rather than forcing the wrapper to parse libvirts output and guess what libvirt does from version to version. For sure, before and after -device, any script which is involved with those options will probably have to change. But that's quite unusual. Most usually I'd expect things like the order of -device options might change depending on how libvirt pulled together it's backing resources, for example. -- Jamie

On 04/12/2010 07:23 AM, Jamie Lokier wrote:

Daniel Veillard wrote:

...

On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:

...

It's not that hard to write this for trivial extra options:

<emulator>/bin/sh -c 'qemu "$0" "$@" -extra-flag'</emulator>

(if that works).

That won't work because we expect the emulator to be a path to the emulator program, so yes that has to be done in the wrapper script.

Ok. Being able to pass arguments would be about infinitely more useful, so that you don't need a separate wrapper script for every individual guest configuration - you can pass options to the single wrapper for each variation.

Even<emulator>/path/to/wrapper qemu-0.11</emulator> is a big improvement over<emulator>/path/to/wrapper-guest-config-6-with-qemu-0.11</emulator> ;-)

Some simple but versatile hook ideas:

-<emulator-append-option> (no space splitting, one option, appended) -<emulator-append-options> (space splitting multiple options) -<emulator-prepend-option> -<emulator-prepend-options> -<emulator-setenv name="NAME">VALUE</emulator-setenv>

I'd strongly suggest not focusing on manipulating command line arguments. I think in the not too distant future, few people will pass command line arguments as opposed to just using configuration files. We are very close to being able to cover 90% of current users via the config file. Regards, Anthony Liguori

-<emulator-wrap>/path/to/script</emulator-wrap> This last one is to call the script, but pass all the options including the path to whichever version of qemu/kvm would get selected by libvirt. (I presume it has some selection mechanism, given the variety of guests which break with each version of qemu and/or qemu-kvm and/or kvm?)

(It is most unfortunate that XML forces those long prefixes everywhere due to lack of contextual naming, unlike less verbose config formats :-/)

...

...

Parsing libvirt output and having to guess which option corresponds to what from the libvirt config sounds very fragile and also a rather large amount of effort for something which should be easy.

it's not that easy because we change the way we build the command line for qemu as qemu evolves. for example before and after the support for -device there have been a lot of changes.

I agree, which is why it's much less fragile if libvirt provides the individual parts to hooks/wrappers, if that's asked, rather than forcing the wrapper to parse libvirts output and guess what libvirt does from version to version.

For sure, before and after -device, any script which is involved with those options will probably have to change. But that's quite unusual. Most usually I'd expect things like the order of -device options might change depending on how libvirt pulled together it's backing resources, for example.

-- Jamie

On Thu, Apr 22, 2010 at 01:47:55PM -0500, Anthony Liguori wrote:

On 04/12/2010 07:23 AM, Jamie Lokier wrote:

...

Daniel Veillard wrote:

...

On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:

...

It's not that hard to write this for trivial extra options:

<emulator>/bin/sh -c 'qemu "$0" "$@" -extra-flag'</emulator>

(if that works).

That won't work because we expect the emulator to be a path to the emulator program, so yes that has to be done in the wrapper script.

Ok. Being able to pass arguments would be about infinitely more useful, so that you don't need a separate wrapper script for every individual guest configuration - you can pass options to the single wrapper for each variation.

Even<emulator>/path/to/wrapper qemu-0.11</emulator> is a big improvement over<emulator>/path/to/wrapper-guest-config-6-with-qemu-0.11</emulator> ;-)

Some simple but versatile hook ideas:

-<emulator-append-option> (no space splitting, one option, appended) -<emulator-append-options> (space splitting multiple options) -<emulator-prepend-option> -<emulator-prepend-options> -<emulator-setenv name="NAME">VALUE</emulator-setenv>

I'd strongly suggest not focusing on manipulating command line arguments. I think in the not too distant future, few people will pass command line arguments as opposed to just using configuration files.

Agreed, we're not having anything in the XML for modifying command line arguments explicitly. In tiny edge case where someone actually needs that, QEMU already provides the '-set OPTION' command line arg that you can use to override any qdev property. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On Fri, 9 Apr 2010 15:27:17 +0100 "Daniel P. Berrange" <berrange@redhat.com> wrote:

On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:

...

Hello, In response to a lot of the talk of qemud lately on qemu-devel, the libvirt community would like to put forward a proposal to help enable debug/advanced options when using various hypervisors. The goals of this API are:

1) To enable more rapid access to hypervisor features before proper libvirt API's are designed around them. 2) To facilitate debugging and access to advanced features that may not fit into the normal libvirt world-view.

Caveats: 1) Unlike other libvirt API's, this one will explicitly *not* be guaranteed ABI/API compatible between libvirt updates.

I think we'd still aim to keep the public API stable. The bit that we can't guarentee is the interactions with QEMU & the libvirt driver. eg, if someone was using the API to send text monitor commands to QEMU, and the next libvirt release switched to JSON mode, that use would break. The API would still be valid, but the way they use it might not be. Similarly if they add some custom extra command line argument, this could potentially conflict with an extra command line arg a subsquent libvirt release used. eg, they used -device to add a PCI card with a specific PCI address. Then next libvirt release specifies this same PCI address and then you get a clash.

Another problem is issuing Monitor commands that could confuse libvirt's real API. Say, adding a device libvirt doesn't know about or stopping the VM while libvirt thinks it's still running or anything like that. When I suggested the monitor passthrough idea, I thought it could be a workaround for the feature time lag problem, but as stated by Avi, I'm afraid that this can potentially cause more problems than it solves. It's very desirable for debugging (item 2 above) and maybe to try new things, but people couldn't use it for anything serious, which is the major issue in this subject (item 1 above). Now, Anthony's idea of having libqemu has the same issues, which seems to mean that the only way to really fix the time lag is to fix the time lag :)

...

Raw access to the qemu monitor will be disabled by default; the <monitorpassthrough/> tag enables the ability to send QMP (or text, if you are using older qemu) messages straight through to the monitor. To do this there will be an additional API entry point named virDomainDebugCommand() which takes an arbitrary string and passes it to the monitor, and returns an arbitrary string as a result. Thus you could pass in either "info cpus" if using the text monitor or '{ "execute": "query-cpus" }' if using QMP.

Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with other hypervisors have different approaches for low level extension/ debug. For example, Xen would involve XenStore access, or XenD XMLRPC, etc. So this should really live in a separate API namespace which is specific to a hypervisor. For example, as a header file

#include <libvirt/libvirt-qemu.h>

Containing APIs like

int virDomainQEMUInvokeMonitor(virDomainPtr dom, const char *command, char **reply);

typedef virConnectQEMUDomainEventCallback(virConnectPtr conn, virDomainPtr dom, const char *eventname, const char *data, void *opaque) int virConnectQEMUDomainEventRegister(virConnectPtr conn, virDomainPtr dom, const char *eventname, virDomainQEMUMonitorCallback cb, void *opaque);

What about all the wonders of json? This will force clients to have their own json <-> text converter, maybe we could provide two APIs, just like qemu driver works internally?

On 04/22/2010 09:49 PM, Anthony Liguori wrote:

...

real API. Say, adding a device libvirt doesn't know about or stopping the VM while libvirt thinks it's still running or anything like that. Another problem is issuing Monitor commands that could confuse libvirt's

We need to make libvirt and qemu smarter.

We already face this problem today with multiple libvirt users. This is why sophisticated management mechanisms (like LDAP) have mechanisms to do transactions or at least a series of atomic operations.

And people said qmp/json was overengineered... But seriously, transactions won't help anything. qemu maintains state, and when you have two updaters touching a shared variable not excepting each other to, things break, no matter how much locking there is. The only way that separate monitors could work is if they touch completely separate state, which is difficult to ensure if you upgrade your libvirt. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.

On 04/23/2010 04:48 PM, Anthony Liguori wrote:

On 04/23/2010 07:48 AM, Avi Kivity wrote:

...

On 04/22/2010 09:49 PM, Anthony Liguori wrote:

...

...

real API. Say, adding a device libvirt doesn't know about or stopping the VM while libvirt thinks it's still running or anything like that. Another problem is issuing Monitor commands that could confuse libvirt's

We need to make libvirt and qemu smarter.

We already face this problem today with multiple libvirt users. This is why sophisticated management mechanisms (like LDAP) have mechanisms to do transactions or at least a series of atomic operations.

And people said qmp/json was overengineered...

But seriously, transactions won't help anything. qemu maintains state, and when you have two updaters touching a shared variable not excepting each other to, things break, no matter how much locking there is.

Let's consider some concrete examples. I'm using libvirt and QMP and in QMP, I want to hot unplug a device.

Today, I do this by listing the pci devices, and issuing a pci_del that takes a PCI address. This is intrinsically racy though because in the worst case scenario, in between when I enumerate pci devices and do the pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add within libvirt of a completely different device.

Obviously you should do the pci_del through libvirt. Once libvirt supports an API, use it.

There are a few ways to solve this, the simplest being that we give devices unique ids that are never reused and instead of pci_del taking a pci bus address, it takes a device id. That would address this race.

You can get very far by just being clever about unique ids and notifications. There are some cases where a true RMW may be required but I can't really think of one off hand. The way LDAP addresses this is that it has a batched operation and a simple set of boolean comparison operations. This lets you execute a batched operation that will do a RMW.

I'm sure we can be very clever, but I'd rather direct this cleverness to qemu core issues, not to the QMP (which in turn requires that users be clever to use it correctly). QMP is a low bandwidth protocol, so races will never show up in testing. We're laying mines here for users to step on that we will never encounter ourselves.

...

The only way that separate monitors could work is if they touch completely separate state, which is difficult to ensure if you upgrade your libvirt.

I don't think this is as difficult of a problem as you think it is. If you look at Active Directory and the whole set of management tools based on it, they certainly allow concurrent management applications. You can certainly get into trouble still but with just some careful considerations, you can make two management applications work together 90% of the time without much fuss on the applications part.

Maybe. We'll still have issues. For example, sVirt: if a QMP command names a labeled resource, the non-libvirt user will have no way of knowing how to label it. Much better to exact a commitment from libvirt to track all QMP (and command line) capabilities. Instead of adding cleverness to QMP, add APIs to libvirt. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Daniel P. Berrange wrote:

...

Much better to exact a commitment from libvirt to track all QMP (and command line) capabilities. Instead of adding cleverness to QMP, add APIs to libvirt.

Agreed. Despite adding this monitor / XML passthrough capability, we still do not want apps to be using this at all. If there is some capability missing that apps need then the default mode of operation is to add the neccessary bits of libvirt. The monitor/XML pasthrough is just a short term quick workaround until the official support is done. As such I do not really think we need to put huge amounts of effort in the wierd complex racey edge cases. The effort is better spent on getting the features in libvirt.

All the features? The qemu API is quite large already (look at all the command line options and monitor commands). I'll be very surprised if libvirt provides all of it that obscure apps may use. I'm thinking of features which are relatively obscure but nonetheless useful to a small number of deployments. Probably not enough to justify the effort building data models, specifying the XML and remote protocol and so on in libvirt. (Unless that becomes so easily mapped to qemu's API that it's almost an automatic thing... Which sounds like QMP, doesn't it?) Is libvirt ever likely to go to the effort of providing all the easily-usable API, or hooks, for: - sending keys to a guest, driven by a timed host script? - rebooting the guest while switching between USB touchpad and mouse devices, because one of them is needed during an OS install and the other is needed after? - changing the amount of RAM available to the guest at the next reboot, for OS install needing more memory than run time, in a scripted fashion when building new VMs from install disk images? - switching the guest between qemu mode and kvm mode on the next guest reset, because qemu is faster for some things (VGA updates) and kvm is faster for other things, so the best choice depends on which app you need to run on that guest - pausing a VM, making a copy, and resuming it, so as to fork it into two VMs (literally fork)? - setting up the host network container and NAT IP forwarding, on demand as guests are stopped and started, so that it works in the above scenario despite clashing IP addresses? - running a copy of the same guest, or perhaps an entire OS install process (scripted), many times for different qemu and qemu-kvm versions, different BIOSes, and different almost-equivalent hardware emulations (i.e. different NIC types, SMP count, CPU features, disk controller type, AIO/cache type) - for testing guests and apps on them - with some paralellism? None of those, except perhaps the first, as what I think of as typical virtualisation workloads, and they all seem obscure things probably outside libvirt's remit. Probably not many users either :-) Yet you can do them all today with qemu and scripting the monitor, and it's getting easier with QMP. Which is fine, qemu works, but it would be great to be able to see those guests and interact in the basic ways through the libvirt-based GUIs? QMP pass-through or QMP multiple monitors seems to provide most of that, although I can see libvirt getting a bit confused about which devices and how much RAM the guest has installed at different times. The bit about forking guests, I'm not sure how complicated it is to tie in to libvirt's notion of which disk images are being used, and hooking into it's network configuration to handle the clashing addresses. If those things are considered to be entirely outside libvirt's remit, that's fine with me. Fair enough: I will continue to live with ssh and vinagre. I'm just raising my hand as a potential user who might like to monitor a bunch of active and inactive guests, remotely, see how much memory they report using, etc. launch VNC viewer from the GUI, even choose the target host based on load and migrate on demand, while also needing a fair bit of non-standardness and qemu-level scripting too. Imho, that probably comes under the heading of apps using pass-through or multiple QMP monitors, which use features that probably won't and probably shouldn't ever be handled by libvirt itself. -- Jamie

On 04/26/2010 05:25 PM, Chris Lalancette wrote:

Right, and you are probably one of the users this work targets. But in general, for those not very familiar with virtualization/qemu, we want to steer them far clear of this API. That goes doubly true for application developers; we want them to be able to use a stable, long-term API and not have to worry about the nitty-gritty details of the monitor. It's that latter group that we want to make sure doesn't use this API.

With qmp, we have a stable long term API, and the nitty-gritty details are easily hidden behind a stock json parser (unfortunately some rpc details remain). The command line is baroque, but the libvirt xml isn't so pretty either. The problem is a user that starts with libvirt and outgrows its featureset. Do we want them to fall back to qmp? -- error compiling committee.c: too many arguments to function

On 04/26/2010 09:54 AM, Daniel P. Berrange wrote:

On Mon, Apr 26, 2010 at 05:34:22PM +0300, Avi Kivity wrote:

...

On 04/26/2010 05:25 PM, Chris Lalancette wrote:

...

Right, and you are probably one of the users this work targets. But in general, for those not very familiar with virtualization/qemu, we want to steer them far clear of this API. That goes doubly true for application developers; we want them to be able to use a stable, long-term API and not have to worry about the nitty-gritty details of the monitor. It's that latter group that we want to make sure doesn't use this API.

With qmp, we have a stable long term API, and the nitty-gritty details are easily hidden behind a stock json parser (unfortunately some rpc details remain). The command line is baroque, but the libvirt xml isn't so pretty either.

The problem is a user that starts with libvirt and outgrows its featureset. Do we want them to fall back to qmp?

If were QMP were to be the standard mgmt API, then you are merely inverting the problem. The problem is now a user that starts with QMP and outgrows its featureset.

We have two layers in the stack here& neither provides a superset of the other. As Anthony has mentioned, this has parallels with GTK (libvirt) vs libX11 (libQMP). Programming against GTK does not give you all the features you'd have if you programmed against X, but most people do not need them and this is offset by the benefits GTK gives in portability across OS. GTK does allow them to get direct access to X if really needed, but most apps will avoid doing that if at all possible, and when it is found to be neccessary new APIs are added to GTK so that direct X11 access remains a short-term hack.

We shouldn't have to frame this as a libvirt vs QMP problem. libvirt uses QMP for its work, and apps use libvirt API for the vast majority of their needs. If direct access to QMP is needed we can now provide that for apps as a short term hack, until suitable APIs are added to libvirt.

I agree with you (for the most part). The one difference here is that libgtk actually implements a full widget kit on top of libX11. That's why people use gtk and not X11 directly. Portability was really an after thought with gtk. If you look at libgtk and libcairo, instead of trying to replicate the features of cairo within gtk, gtk encourages people to use cairo directly for anything beyond the simplest 2d operations. Regards, Anthony Liguori

Daniel

On 04/26/2010 10:20 AM, Daniel P. Berrange wrote:

For the sake of my analogy, I'd still group cairo in with gtk, not x11. It is really just a re-implmentation of the GDK drawing layer from GTK. It still provides a portable higher level abstraction over underlying graphics & rendering systems it supports. eg, again an app might do the majority of its drawing with Cairo, but occasionally need to jump outside to use XRender or OpenGL directly for pieces that aren't supported currently in the cairo API.

Yeah, that's a fair point. Regards, Anthony Liguori

Regards, Daniel

On 04/23/2010 09:29 PM, Anthony Liguori wrote:

...

Maybe. We'll still have issues. For example, sVirt: if a QMP command names a labeled resource, the non-libvirt user will have no way of knowing how to label it.

This is orthogonal to QMP and has to do strictly with how libvirt prepares a resource for qemu.

It's not orthogonal. If you allow qmp access behind libvirt's back, it's a problem that you will have.

...

Much better to exact a commitment from libvirt to track all QMP (and command line) capabilities. Instead of adding cleverness to QMP, add APIs to libvirt.

Let's step back for a minute because I think we're missing the forest through the trees.

We're trying to address a few distinct problems:

1) Allow libvirt users to access features of qemu that are not exposed through libvirt

That's an artificial problem. If libvirt exposes all features, you don't need to solve it.

2) Provide a means for non-libvirt users to interact with qemu

We have qmp. It doesn't do multiple guest management. I think it's reasonable to have a qemud which does (and also does sVirt and the zillion other things libvirt does) provided we remove them from libvirt (long term). The only problem is that it's a lot of effort.

3) Provide a unified and interoperable view of the world for non-libvirt and libvirt users

This problem can be solved by the non-libvirt users adopting libvirt, or the libvirt users dropping libvirt. I don't understand why we need to add interoperability between users who choose an interoperability library and users who don't choose an interoperability library.

For (1), we all agree that the best case scenario would be for libvirt to support every qemu feature. I think we can also all agree though that this is not really practical and certainly not practical for developers since there is a development cost associated with libvirt support (to model an API appropriately).

All except me, perhaps. We already have two layers of feature modeling: first, we mostly emulate real life, not invent new features. PCI hotplug existed long before qemu had support for it. Second, we do give some thought into how we expose it through QMP. libvirt doesn't have to invent it again, it only has to expose it through its lovely xml and C APIs.

The new API proposed addresses (1) by allowing a user to drill down to the QMP context. It's a good solution IMHO and I think we all agree that there's an inherent risk to this that users will have to evaluate on a case-by-case basis. It's a good stop-gap though.

Agree.

(2) is largely addressed by QMP and a config file. I'd like to see a nice C library, but I think a lot of other folks are happy with JSON support in higher level languages.

I agree with them. C is a pretty bad choice for managing qemu (or even, C is a pretty bad choice).

(3) is the place where there are still potential challenges. I think at the very least, our goal should be to enable conversion from (2) and (1) to be as easy as possible. That's why I have proposed implementing a C library for the JSON transport because we could plumb that through the new libvirt API. This would allow a user to very quickly port an application from QMP to libvirt. In order to do this, we need the libvirt API to expose a dedicated monitor because we'll need to be able to manipulate events and negotiate features.

Most likely any application that talks QMP will hide the protocol behind a function call interface anyway.

Beyond simple porting, there's a secondary question of having non-libvirt apps co-exist with libvirt apps. I think it's a good long term goal, but I don't think we should worry too much about it now.

libvirt needs to either support all but the most esoteric use cases, or to get out of the way completely. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.

On 04/25/2010 06:39 AM, Anthony Liguori wrote:

On 04/24/2010 04:46 AM, Avi Kivity wrote:

...

On 04/23/2010 09:29 PM, Anthony Liguori wrote:

...

...

Maybe. We'll still have issues. For example, sVirt: if a QMP command names a labeled resource, the non-libvirt user will have no way of knowing how to label it.

This is orthogonal to QMP and has to do strictly with how libvirt prepares a resource for qemu.

It's not orthogonal. If you allow qmp access behind libvirt's back, it's a problem that you will have.

My point was, if libvirt is just exposing raw qemu features, then it should be possible for qemu to arbitrate concurrent access. If libvirt implements features on top of qemu, then no other third party will be able to co-exist with those features without interacting with qemu. It's an impossible problem for qemu to solve (arbitrating access to state stored in a third party management app).

If libvirt implement features (like sVirt or network configuration) then it is indeed impossible for qemu to arbitrate. If we take all those features into qemu[d], then it becomes possible to arbitrate so long as the libvirt and the other management app don't step on each others toes. But that's impossible to guarantee if you upgrade your libvirt while keeping the other app unchanged.

...

...

1) Allow libvirt users to access features of qemu that are not exposed through libvirt

That's an artificial problem. If libvirt exposes all features, you don't need to solve it.

It won't. Otherwise, we wouldn't be having this discussion.

Then libvirt will fade into uselessness. A successful app using libvirt will grow, and will have new requirements. As soon as libvirt doesn't meet those new requirements, the app will need to talk to qemu[d] directly. Once it does that, it may as well use qemu[d] for everything; if you can talk QMP and generate qemu command lines, there's not much that libvirt buys you. Even the cross-hypervisor support is not that hard to implement, especially if you only need to satisfy your own requirements.

...

...

2) Provide a means for non-libvirt users to interact with qemu

We have qmp. It doesn't do multiple guest management. I think it's reasonable to have a qemud which does (and also does sVirt and the zillion other things libvirt does) provided we remove them from libvirt (long term). The only problem is that it's a lot of effort.

It depends on what things you think are important. A lot of libvirt's complexity is based on the fact that it uses a daemon and needs to deal with the security implications of that. You don't need explicit labelling if you don't use a daemon.

I don't follow. If you have multiple guests that you want off each other's turf you have to label their resources, either statically or dynamically. How is it related to a daemon being present?

This is really the qemu model (as opposed to the xend model).

(and the qemud model).

In theory, it does support this with the session urls but they are currently second-class citizens in libvirt. The remote dispatch also adds a fair bit of complexity and at least for the use-cases I'm interested in, it's not an important feature.

If libvirt needs a local wrapper for interesting use cases, then it has failed. You can't have a local wrapper with the esx driver, for example. This is off-topic, but can you detail why you don't want remote dispatch (I assume we're talking about a multiple node deployment).

...

...

3) Provide a unified and interoperable view of the world for non-libvirt and libvirt users

This problem can be solved by the non-libvirt users adopting libvirt, or the libvirt users dropping libvirt. I don't understand why we need to add interoperability between users who choose an interoperability library and users who don't choose an interoperability library.

What I'd like to avoid is user confusion. Should a user use libvirt or libqemu? If they make a decision to use libqemu and then down the road want to use libvirt, how hard is it to switch? Fragmentation hurts the ecosystem and discourages good applications from existing. I think it's our responsibility to ensure there's a good management API that exists for qemu that we can actively recommend to our users. libvirt is very good at typical virtualization uses of qemu but qemu is much more than just that and has lots of advanced features.

Every typical virtualization use will eventually grow some non-typical requirements. If libvirt explicitly refuses to support qemu features, I don't see how we can recommend it - even if it satisfies a user's requirements today, what about tomorrow? what about future qemu feature, will they be exposed or not? If that is the case then we should develop qemud (which libvirt and other apps can use). (even if it isn't the case I think qemud is a good idea)

...

...

For (1), we all agree that the best case scenario would be for libvirt to support every qemu feature. I think we can also all agree though that this is not really practical and certainly not practical for developers since there is a development cost associated with libvirt support (to model an API appropriately).

All except me, perhaps.

We already have two layers of feature modeling: first, we mostly emulate real life, not invent new features. PCI hotplug existed long before qemu had support for it. Second, we do give some thought into how we expose it through QMP. libvirt doesn't have to invent it again, it only has to expose it through its lovely xml and C APIs.

That's not what the libvirt community wants to do. We're very bias. We've made decisions about how features should be exposed and what features should be included. We want all of those features exposed exactly how we've implemented them because we think it's the right way.

I'm not sure there's an obvious way forward unless we decide that there is going to be two ways to interact with qemu. One way is through the libvirt world-view and the other is through a more qemu centric view. The problem then becomes allowing those two models to co-exist happily together.

I don't think there's a point in managing qemu through libvirt and directly in parallel. It means a user has to learn both APIs, and for every operation they need to check both to see what's the best way of exploiting the feature. There will invariably be some friction. Layers need to stack on top of each other, not live side by side or bypass each other.

The alternative is to get libvirt to just act as a thin layer to expose qemu features directly. But honestly, what's the point of libvirt if they did that?

For most hypervisors, that's exactly what libvirt does. For Xen, it also bypasses Xend and the hypervisor's API, but it shouldn't really. Qemu is special due to the nonexistence of qemud. Why is sVirt implemented in libvirt? it's not the logical place for it; rather the logical place doesn't exist.

For the most part, the pool management is not virtualization specific and you could have libvirt provide that functionality without it knowing a thing about qemu.

Yes.

...

...

Beyond simple porting, there's a secondary question of having non-libvirt apps co-exist with libvirt apps. I think it's a good long term goal, but I don't think we should worry too much about it now.

libvirt needs to either support all but the most esoteric use cases, or to get out of the way completely.

That's not where it is today or the path I think they're taking. I'm also not the person you have to convince of this.

It's fun trying all the same. But you are the person to convince that side-by-side management is doomed to fail, yes? -- error compiling committee.c: too many arguments to function

On 04/26/2010 04:53 AM, Anthony Liguori wrote:

On 04/25/2010 06:51 AM, Avi Kivity wrote:

...

...

It depends on what things you think are important. A lot of libvirt's complexity is based on the fact that it uses a daemon and needs to deal with the security implications of that. You don't need explicit labelling if you don't use a daemon.

I don't follow. If you have multiple guests that you want off each other's turf you have to label their resources, either statically or dynamically. How is it related to a daemon being present?

Because libvirt has to perform this labelling because it loses the original user's security context.

If you invoke qemu with the original user's credentials that launched the guest, then you don't need to do anything special with respect to security.

IOW, libvirt does not run guests as separate users which is why it needs to deal with security in the first place.

What if one user has multiple guests? isolation is still needed. One user per guest does not satisfy some security requirements. The 'M' in selinux stands for mandatory, which means that the entities secured can't leak information even if they want to (scenario: G1 breaks into qemu, chmods files, G2 breaks into qemu, reads files).

...

...

This is really the qemu model (as opposed to the xend model).

(and the qemud model).

And I've said in the past that I don't like the idea of a qemud :-)

I must have missed it. Why not? Every other hypervisor has a central management entity.

...

...

In theory, it does support this with the session urls but they are currently second-class citizens in libvirt. The remote dispatch also adds a fair bit of complexity and at least for the use-cases I'm interested in, it's not an important feature.

If libvirt needs a local wrapper for interesting use cases, then it has failed. You can't have a local wrapper with the esx driver, for example.

This is off-topic, but can you detail why you don't want remote dispatch (I assume we're talking about a multiple node deployment).

Because there are dozens of remote management APIs and then all have a concept of agents that run on the end nodes. When fitting virtualization management into an existing management infrastructure, you are going to always use a local API.

When you manage esx, do you deploy an agent? I thought it was all done via their remote APIs.

...

Every typical virtualization use will eventually grow some non-typical requirements. If libvirt explicitly refuses to support qemu features, I don't see how we can recommend it - even if it satisfies a user's requirements today, what about tomorrow? what about future qemu feature, will they be exposed or not?

If that is the case then we should develop qemud (which libvirt and other apps can use).

(even if it isn't the case I think qemud is a good idea)

Yeah, that's where I'm at. I'd eventually like libvirt to use our provided API and I can see where it would add value to the stack (by doing things like storage and network management).

We do provide an API, qmp, and libvirt uses it?

...

...

That's not what the libvirt community wants to do. We're very bias. We've made decisions about how features should be exposed and what features should be included. We want all of those features exposed exactly how we've implemented them because we think it's the right way.

I'm not sure there's an obvious way forward unless we decide that there is going to be two ways to interact with qemu. One way is through the libvirt world-view and the other is through a more qemu centric view. The problem then becomes allowing those two models to co-exist happily together.

I don't think there's a point in managing qemu through libvirt and directly in parallel. It means a user has to learn both APIs, and for every operation they need to check both to see what's the best way of exploiting the feature. There will invariably be some friction.

Layers need to stack on top of each other, not live side by side or bypass each other.

I agree with you theoretically but practically, I think it's immensely useful as a stop-gap.

Sure. But please lets not start being clever with transactions and atomic operations and stuff, it has to come with a label that says, if you're using this, then something is wrong.

...

...

The alternative is to get libvirt to just act as a thin layer to expose qemu features directly. But honestly, what's the point of libvirt if they did that?

For most hypervisors, that's exactly what libvirt does. For Xen, it also bypasses Xend and the hypervisor's API, but it shouldn't really.

Historically, xend was so incredibly slow (especially for frequent statistics collection) that it was a necessity.

Ah, reimplement rather than fix.

...

Qemu is special due to the nonexistence of qemud.

Why is sVirt implemented in libvirt? it's not the logical place for it; rather the logical place doesn't exist.

sVirt is not just implemented in libvirt. libvirt implements a mechanism to set the context of a given domain and dynamically label it's resources to isolate it.

The reason it has to assign a context to a given domain is that all domains are launched from the same security context (the libvirtd context) as the original user's context (the consumer of the libvirt API) has been lost via the domain socket interface.

If you used the /session URL, then the domain would have the security context of whomever created the guest which means that dynamic labelling of the resources wouldn't be necessary (you would just do static labelling).

This is certainly a more secure model and it's a feature of qemu that I really wish didn't get lost in libvirt. Again, /session can do this too but right now, /session really isn't usable in libvirt for qemu.

That's wrong for three reasons. First, selinux is not a uid replacement (if it was libvirt could just suid $random_user before launching qemu). Second, a single user's guests should be protected from each other. Third, in many deployments, the guest's owner isn't logged in to supply the credentials, it's system management that launches the guests. There's also the case of resources that can't be permanently chowned or assigned a security label, like disk volumes or assignable devices. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.

On 04/26/2010 12:56 AM, Avi Kivity wrote:

On 04/26/2010 04:53 AM, Anthony Liguori wrote:

...

On 04/25/2010 06:51 AM, Avi Kivity wrote:

...

...

It depends on what things you think are important. A lot of libvirt's complexity is based on the fact that it uses a daemon and needs to deal with the security implications of that. You don't need explicit labelling if you don't use a daemon.

I don't follow. If you have multiple guests that you want off each other's turf you have to label their resources, either statically or dynamically. How is it related to a daemon being present?

Because libvirt has to perform this labelling because it loses the original user's security context.

If you invoke qemu with the original user's credentials that launched the guest, then you don't need to do anything special with respect to security.

IOW, libvirt does not run guests as separate users which is why it needs to deal with security in the first place.

What if one user has multiple guests? isolation is still needed.

Don't confuse a management application's concept of users with using separate uid's to launch guests.

One user per guest does not satisfy some security requirements. The 'M' in selinux stands for mandatory, which means that the entities secured can't leak information even if they want to (scenario: G1 breaks into qemu, chmods files, G2 breaks into qemu, reads files).

If you're implementing a chinese firewall policy, then yes, you want to run each guest as a separate selinux context. Starting as separate users and setting DAC privileges appropriately will achieve this. But you're not always implementing that type of policy. If the guest inherits the uid, selinux context, and namespaces of whatever launches the guest, then you have the most flexibility from a security perspective. How do you launch a libvirt guest in a network namespace? How do you put it in a chroot? Today, you have to make changes to libvirt whereas in a direct launch model, you get all of the neat security features linux supports for free.

...

And I've said in the past that I don't like the idea of a qemud :-)

I must have missed it. Why not? Every other hypervisor has a central management entity.

Because you end up launching all guests from a single security context.

...

...

...

In theory, it does support this with the session urls but they are currently second-class citizens in libvirt. The remote dispatch also adds a fair bit of complexity and at least for the use-cases I'm interested in, it's not an important feature.

If libvirt needs a local wrapper for interesting use cases, then it has failed. You can't have a local wrapper with the esx driver, for example.

This is off-topic, but can you detail why you don't want remote dispatch (I assume we're talking about a multiple node deployment).

Because there are dozens of remote management APIs and then all have a concept of agents that run on the end nodes. When fitting virtualization management into an existing management infrastructure, you are going to always use a local API.

When you manage esx, do you deploy an agent? I thought it was all done via their remote APIs.

Historically, people have deployed agents into the console OS. In recent versions, ESX actually includes CIM agents by default.

...

...

Every typical virtualization use will eventually grow some non-typical requirements. If libvirt explicitly refuses to support qemu features, I don't see how we can recommend it - even if it satisfies a user's requirements today, what about tomorrow? what about future qemu feature, will they be exposed or not?

If that is the case then we should develop qemud (which libvirt and other apps can use).

(even if it isn't the case I think qemud is a good idea)

Yeah, that's where I'm at. I'd eventually like libvirt to use our provided API and I can see where it would add value to the stack (by doing things like storage and network management).

We do provide an API, qmp, and libvirt uses it?

Yeah, but we need to support more features (like guest enumeration).

...

...

...

The alternative is to get libvirt to just act as a thin layer to expose qemu features directly. But honestly, what's the point of libvirt if they did that?

For most hypervisors, that's exactly what libvirt does. For Xen, it also bypasses Xend and the hypervisor's API, but it shouldn't really.

Historically, xend was so incredibly slow (especially for frequent statistics collection) that it was a necessity.

Ah, reimplement rather than fix.

There's a complicated history there.

...

...

Qemu is special due to the nonexistence of qemud.

Why is sVirt implemented in libvirt? it's not the logical place for it; rather the logical place doesn't exist.

sVirt is not just implemented in libvirt. libvirt implements a mechanism to set the context of a given domain and dynamically label it's resources to isolate it.

The reason it has to assign a context to a given domain is that all domains are launched from the same security context (the libvirtd context) as the original user's context (the consumer of the libvirt API) has been lost via the domain socket interface.

If you used the /session URL, then the domain would have the security context of whomever created the guest which means that dynamic labelling of the resources wouldn't be necessary (you would just do static labelling).

This is certainly a more secure model and it's a feature of qemu that I really wish didn't get lost in libvirt. Again, /session can do this too but right now, /session really isn't usable in libvirt for qemu.

That's wrong for three reasons. First, selinux is not a uid replacement (if it was libvirt could just suid $random_user before launching qemu). Second, a single user's guests should be protected from each other. Third, in many deployments, the guest's owner isn't logged in to supply the credentials, it's system management that launches the guests.

(1) uid's are just one part of an applications security context. There's an selinux context, all of the various namespaces, capabilities, etc. If you use a daemon to launch a guest, you lose all of that unless you have a very sophisticated api. (2) If you want to implement a policy that only a single guest can access a single image, you can create an SELinux policy and use static labelling to achieve that. That's just one type of policy though. (3) The system management application can certainly create whatever context it wants to launch a vm from. It's comes down to who's responsible for creating the context the guest runs under. I think doing that at the libvirt level takes away a ton of flexibility from the management application. Regards, Anthony Liguori

There's also the case of resources that can't be permanently chowned or assigned a security label, like disk volumes or assignable devices.

On 04/26/2010 08:41 AM, Avi Kivity wrote:

...

Today, you have to make changes to libvirt whereas in a direct launch model, you get all of the neat security features linux supports for free.

But you lose tap networking, unless you have a privileged helper. And how is the privileged helper to authenticate the qemu calling it?

There are a variety of ways. My original proposal used a policy file.

...

...

...

And I've said in the past that I don't like the idea of a qemud :-)

I must have missed it. Why not? Every other hypervisor has a central management entity.

Because you end up launching all guests from a single security context.

Run multiple qemuds?

But what you say makes sense. It's similar to the fork() /* do interesting stuff */ exec() model, compared to the spawn(..., hardcoded list of interesting stuff).

...

...

...

Yeah, that's where I'm at. I'd eventually like libvirt to use our provided API and I can see where it would add value to the stack (by doing things like storage and network management).

We do provide an API, qmp, and libvirt uses it?

Yeah, but we need to support more features (like guest enumeration).

What are our options?

1) qemud launches, enumerates 2) user launches, qemu registers in qemud 3) user launches, qemu registers in filesystem 4) you launched it, you enumerate it

Both 2 and 3 are appealing to me.

...

(3) The system management application can certainly create whatever context it wants to launch a vm from. It's comes down to who's responsible for creating the context the guest runs under. I think doing that at the libvirt level takes away a ton of flexibility from the management application.

If you want to push the flexibility slider all the way to the right you get bare qemu. It exposes 100% of qemu capabilities. And it's not so bad these days. But it's not something that can be remoted.

As I mentioned earlier, remoting is not a very important use-case to me. Does RHEV-M actually use the remote libvirt interface? I assume it'll talk to vdsm via some protocol and vdsm will use the local libvirt API. I suspect most uses of libvirt are actually local uses. Regards, Anthony Liguori

On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote:

On 04/26/2010 08:41 AM, Avi Kivity wrote:

...

...

(3) The system management application can certainly create whatever context it wants to launch a vm from. It's comes down to who's responsible for creating the context the guest runs under. I think doing that at the libvirt level takes away a ton of flexibility from the management application.

If you want to push the flexibility slider all the way to the right you get bare qemu. It exposes 100% of qemu capabilities. And it's not so bad these days. But it's not something that can be remoted.

As I mentioned earlier, remoting is not a very important use-case to me.

NB, the "remote" protocol is no different from the "local" protocol the unprivileged app uses to talk to the privileged daemon. The only difference is unix sockets vs tcp sockets. We essentially get the remote access capability for free as part of the protocol used between the apps & daemon.

Does RHEV-M actually use the remote libvirt interface? I assume it'll talk to vdsm via some protocol and vdsm will use the local libvirt API.

That is correct. That doesn't mean the remote interface isn't useful.

I suspect most uses of libvirt are actually local uses.

A key part of libvirt is interoperability between applications. So an app like vsdm can be using libvirt locally, while at the same time virt-top can be monitoring guests on a set of machines remotely, or virt-viewer can be run from your desktop talking to libvirt remotely to discover the VNC console details. For flexibility there are several options for remotely managing VMs. The native remote RPC system. The libvirt QPid/QMF model. The CIM binding. Application specific remote protocols. Since they all eventually talk to the same libvirtd daemon, their use is not mutually exclusive which is a good thing. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On Mon, Apr 26, 2010 at 09:26:55AM -0500, Anthony Liguori wrote:

On 04/26/2010 08:58 AM, Daniel P. Berrange wrote:

...

On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote:

...

On 04/26/2010 08:41 AM, Avi Kivity wrote:

...

...

(3) The system management application can certainly create whatever context it wants to launch a vm from. It's comes down to who's responsible for creating the context the guest runs under. I think doing that at the libvirt level takes away a ton of flexibility from the management application.

If you want to push the flexibility slider all the way to the right you get bare qemu. It exposes 100% of qemu capabilities. And it's not so bad these days. But it's not something that can be remoted.

As I mentioned earlier, remoting is not a very important use-case to me.

NB, the "remote" protocol is no different from the "local" protocol the unprivileged app uses to talk to the privileged daemon. The only difference is unix sockets vs tcp sockets. We essentially get the remote access capability for free as part of the protocol used between the apps& daemon.

I think Avi's concern is that if you expose direct launch, then that's not really something you can usefully exploit in a remote protocol.

For instance, if you're local, you can call setuid() before invoking the launch API whereas if you're remote, you will always launch guests as the context of the remoting daemon.

This makes the remote API second-class in terms of functionality.

It is not entirely outside the realms of possibility to consider making a way for libvirt to attach to a QEMU spawned by a 3rd party. eg, virsh qemu-attach /path/to/monitor PID Converting the QEMU config back into Libvirt XML is not trivial, but neither is it impossible. Particularly now we have the custom <qemu:arg> syntax for things libvirt can't represent, we can simply use that for bits we can't reverse map to XML. There's also the UUID/name uniqueness, but we could fail the attach operation if that wwas violated. This would let you use most of the other libvirt APIs remotely from any app, even if the VM was spawned by another one. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On 04/26/2010 04:59 AM, Daniel P. Berrange wrote:

On Sun, Apr 25, 2010 at 08:53:17PM -0500, Anthony Liguori wrote:

...

On 04/25/2010 06:51 AM, Avi Kivity wrote:

...

Qemu is special due to the nonexistence of qemud.

Why is sVirt implemented in libvirt? it's not the logical place for it; rather the logical place doesn't exist.

sVirt is not just implemented in libvirt. libvirt implements a mechanism to set the context of a given domain and dynamically label it's resources to isolate it.

The reason it has to assign a context to a given domain is that all domains are launched from the same security context (the libvirtd context) as the original user's context (the consumer of the libvirt API) has been lost via the domain socket interface.

If you used the /session URL, then the domain would have the security context of whomever created the guest which means that dynamic labelling of the resources wouldn't be necessary (you would just do static labelling).

That is not correct. You do *not* ever want the guests to have the same security context as the thing that created them, because that would allow the guest to access& compromise resources belonging to the management app.

You assume that the management app is not smart enough to create a new context for the guest to run in.

...

This is certainly a more secure model and it's a feature of qemu that I really wish didn't get lost in libvirt. Again, /session can do this too but right now, /session really isn't usable in libvirt for qemu.

If you really want the qemu instance to inherit the context of the mgmt app, then you can just declare in the guest XML that it should use a static label, and pass in the apps' own label. This is *not* a more secure model though.

There is more context than just selinux labelling. The problem with the daemon model is that to create a guest, you start with a lower set of privileges, escalate your privileges (by talking to libvirtd), then lower privileges to launch a guest. Running a guest is essentially running arbitrary code (since you can set the emulator path) so now you've provided an environment where a user can launch arbitrary code as a different user in a different security context. There is a new attack surface here. I think it's undeniable that there is certainly the possibility that something goes wrong and a user will find a way to escalate it's privileges. Compare that to a direct launch model. There is not new attack surface. The user's privileges never increase. In fact, what's most likely to happen is that a caller will drop some of it's privileges before launching a guest. Regards, Anthony Liguori

Daniel

On 04/26/2010 08:31 AM, Daniel P. Berrange wrote:

What you describe is not inherant to the daemon model. This is why we have two separate models in libvirt. The system instance is pre-spawned with high privileges, to allow use of hosts resources which require high privileges to access. The session instance is auto-spawned when the app connects to libvirt, thus it inherits the privileges of the app that is using it.

I don't deny that the system instance has a new attack surface, because it is running privileged. If the app needs to connect VMs to privileged resources, then the architecture has to have some privileged component to give access to those resoruces. You don't want the VM to be privileged, nor the whole management app to be privileged. The system daemon is thus the arbitrator for this privileged access. If you don't need todo anything that requires higher privileges though, just use the session instance which always matches the apps privileges.

I regret saying "more secure" because I think it's a difficult concept to really quantify and that makes it hard to meaningfully discuss. The reason I lean toward the direct launch model is that it gives the user a lot of flexibility in terms of using things like namespaces, DAC, cgroups, capabilities, etc. A lot of potential features are lost when you do indirect launch because you have to teach the daemon how to support each of these features. Regards, Anthony Liguori

Daniel

On 04/26/2010 09:01 AM, Avi Kivity wrote:

On 04/26/2010 04:43 PM, Anthony Liguori wrote:

...

The reason I lean toward the direct launch model is that it gives the user a lot of flexibility in terms of using things like namespaces, DAC, cgroups, capabilities, etc. A lot of potential features are lost when you do indirect launch because you have to teach the daemon how to support each of these features.

But what's the alternative? Teach the user how to do all these things?

You can expose layers of API. The lowest layer makes no changes to the security context. A higher (optional) layer could do dynamic labelling.

It's infinitely flexible, but it's not an API you can give to a management tool developer.

I think the goal of a management API should be to make common things very simple to do but not preclude doing even the most advanced things. Regards, Anthony Liguori

On 04/26/2010 09:25 AM, Avi Kivity wrote:

On 04/26/2010 05:19 PM, Anthony Liguori wrote:

...

On 04/26/2010 09:01 AM, Avi Kivity wrote:

...

On 04/26/2010 04:43 PM, Anthony Liguori wrote:

...

The reason I lean toward the direct launch model is that it gives the user a lot of flexibility in terms of using things like namespaces, DAC, cgroups, capabilities, etc. A lot of potential features are lost when you do indirect launch because you have to teach the daemon how to support each of these features.

But what's the alternative? Teach the user how to do all these things?

You can expose layers of API. The lowest layer makes no changes to the security context. A higher (optional) layer could do dynamic labelling.

Or a library that the user-written launcher calls. Or a plugin that qemud calls.

A plugin would lose the security context. It could attempt to recreate it that seems like a lot of unnecessary complexity. Regards, Anthony Liguori

On 04/26/2010 09:38 AM, Avi Kivity wrote:

On 04/26/2010 05:28 PM, Anthony Liguori wrote:

...

...

Or a library that the user-written launcher calls. Or a plugin that qemud calls.

A plugin would lose the security context. It could attempt to recreate it that seems like a lot of unnecessary complexity.

A plugin would create the security context instead of the launcher.

Currently security contexts are created by the login process.

It's not always that centralized. An initial context is created by the login process, but then later something may come along and create a network namespace as part of containerization.

We could easily reuse that. Any other security context code would be custom written; so it can be written as a qemud plugin instead of a bit of code that goes before a qemu launch.

I think we're mostly in agreement with respect to the need to have more control over the security context the qemu runs in. Whether it's launched via a daemon or directly I think is an implementation detail that we can debate when we get closer to an actual implementation. Regards, Anthony Liguori

On Fri, Apr 23, 2010 at 08:48:51AM -0500, Anthony Liguori wrote:

On 04/23/2010 07:48 AM, Avi Kivity wrote:

...

On 04/22/2010 09:49 PM, Anthony Liguori wrote:

...

...

real API. Say, adding a device libvirt doesn't know about or stopping the VM while libvirt thinks it's still running or anything like that. Another problem is issuing Monitor commands that could confuse libvirt's

We need to make libvirt and qemu smarter.

We already face this problem today with multiple libvirt users. This is why sophisticated management mechanisms (like LDAP) have mechanisms to do transactions or at least a series of atomic operations.

And people said qmp/json was overengineered...

But seriously, transactions won't help anything. qemu maintains state, and when you have two updaters touching a shared variable not excepting each other to, things break, no matter how much locking there is.

Let's consider some concrete examples. I'm using libvirt and QMP and in QMP, I want to hot unplug a device.

Today, I do this by listing the pci devices, and issuing a pci_del that takes a PCI address. This is intrinsically racy though because in the worst case scenario, in between when I enumerate pci devices and do the pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add within libvirt of a completely different device.

This is what already happens with any QEMU >= 0.12, where libvirt uses the new -device syntax with its 'id' parameter for all devices, and then uses 'device_id $ID' for unplug. The app still has to be careful it doesn't try to add a device using the same naming scheme as a device libvirt uses. This is easy enough though if they prepend some random string to all device IDs thy use, that won't clash with libvirt device ID names. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:

On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:

...

Hello, In response to a lot of the talk of qemud lately on qemu-devel, the libvirt community would like to put forward a proposal to help enable debug/advanced options when using various hypervisors. The goals of this API are:

1) To enable more rapid access to hypervisor features before proper libvirt API's are designed around them. 2) To facilitate debugging and access to advanced features that may not fit into the normal libvirt world-view.

Caveats: 1) Unlike other libvirt API's, this one will explicitly *not* be guaranteed ABI/API compatible between libvirt updates.

I think we'd still aim to keep the public API stable. The bit that we can't guarentee is the interactions with QEMU& the libvirt driver. eg, if someone was using the API to send text monitor commands to QEMU, and the next libvirt release switched to JSON mode, that use would break. The API would still be valid, but the way they use it might not be. Similarly if they add some custom extra command line argument, this could potentially conflict with an extra command line arg a subsquent libvirt release used. eg, they used -device to add a PCI card with a specific PCI address. Then next libvirt release specifies this same PCI address and then you get a clash.

So in both cases the API and XML format can be reasonably guarenteed between releases. What we can't guarentee is that usage of these features will be reliable across releases of libvirt.

...

2) Again unlike other libvirt API's, access and configuration of the debug section of a domain will be highly hypervisor dependent. 3) Application developers will be strongly discouraged from using this API because of the above 2 issues. To help in this, the API's will be in a separate library that developers will explicitly have to link to, and it will have a different (but largely compatible) wire protocol.

In terms of wire protocol, this would still have to run over the same existing data channel, because we don't want apps to have to open up multiple connections. Fortunately our protocol is designed to allow this kind of extension

struct remote_message_header { unsigned prog; /* REMOTE_PROGRAM */ unsigned vers; /* REMOTE_PROTOCOL_VERSION */ remote_procedure proc; /* REMOTE_PROC_x */ remote_message_type type; unsigned serial; /* Serial number of message. */ remote_message_status status; };

We currently have a single program 'REMOTE_PROGRAM' and its associated 'vers' and 'proc' numbers. What we would be doing is to define a new program, say QEMU_PROGRAM. The 'remote_procedure proc' numbers can thus start again from '1' without clashing with any of the existing APIs. All the RPC marshalling/demarshalling code should work more or less unchanged. We just hook in a different function dispatch table in the daemon for that program.

...

4) We don't expect this API to solve all of the issues brought up during the qemud discussion. Our initial goal is just to give ready access of the qemu command-line and monitor to developers.

With that being said, our initial proposal follows. We expect this to evolve over time as we get more feedback, but we think this proposal addresses at least 2 of the major pain points qemu developers have while trying to use libvirt.

The initial debug XML for qemu would be:

I wouldn't call them 'debug' options, since that's just one of possible use cases for this. What we're really doing there is exposing hypervisor specific features, so we should just be upfront about that in our description / naming.

...

<domain type='kvm'> <name>myguest</name> ... <debug> <monitorpassthrough/> <commandline> <extra>qemu arguments</extra> <alter option="optname"> <rename>newname</rename> <match>REGEXP</match> <modify>foo=on</modify> <extra>-bar</extra> </alter> </commandline> </debug> </domain>

The concept of command line& monitor is something that is QEMU specific and thus is not suitable for the primary XML schema. IMHO, this needs to be done as a separate schema, linked in via an XML namespace. For example

<domain type='kvm' xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0"> <name>myguest</name> ... <qemu:commandline> <qemu:arg>-device</arg> <qemu:arg>lsi</arg> </qemu:commandline> </domain>

I think it's problematic to focus too much on command line arguments. We are not introducing new command line arguments to qemu for the most part that aren't usable in the config file. What I really think we would like, is a way to express the qemu configuration file in terms of XML and then to use snippets of that within an XML namespace. For instance: [cpudef] name = "Opteron_G3" level = "5" vendor = "AuthenticAMD" ... Could convert to: <cpudef> <name>Operon_G3</name> <level>5</level> <vendor>AuthenticAMD</vendor> ... </cpudef> Which in turn, could be embedded as: <domain type='kvm' xmlns:qemu='http://qemu.org/schemas/qemu/1.0'> <name>myguest</name> ... <qemu:cpudef> <name>Operon_G3</name> <level>5</level> <vendor>AuthenticAMD</vendor> </qemu:cpudef> </domain> With respect to injecting QMP commands directly, I think the proposed debug API is probably reasonable. We could build a libqemu that used that API as a transport which means that one could use libqemu and libvirt simultaneously which is certainly a key requirement of mine. I think it's important that it's a dedicated monitor session though. It shouldn't just be injecting commands within an existing QMP session IMHO. Regards, Anthony Liguori Regards, Anthony Liguori

On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote:

On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:

...

On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:

...

<domain type='kvm'> <name>myguest</name> ... <debug> <monitorpassthrough/> <commandline> <extra>qemu arguments</extra> <alter option="optname"> <rename>newname</rename> <match>REGEXP</match> <modify>foo=on</modify> <extra>-bar</extra> </alter> </commandline> </debug> </domain>

The concept of command line& monitor is something that is QEMU specific and thus is not suitable for the primary XML schema. IMHO, this needs to be done as a separate schema, linked in via an XML namespace. For example

<domain type='kvm' xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0"> <name>myguest</name> ... <qemu:commandline> <qemu:arg>-device</arg> <qemu:arg>lsi</arg> </qemu:commandline> </domain>

I think it's problematic to focus too much on command line arguments. We are not introducing new command line arguments to qemu for the most part that aren't usable in the config file.

Currently libvirt doesn't use any config file at all, so command line is the only possible option. We're going to evaluate switching to the config file at some point & so can easily add further XML options to allow direct setting of config file entries at that point. There's no problem supporting both command line & config file syntax in the XML. The command line does have the immediate advantage that 99% of all documentation about QEMU on the web today refers to command line args so that's what people are most familiar with.

With respect to injecting QMP commands directly, I think the proposed debug API is probably reasonable. We could build a libqemu that used that API as a transport which means that one could use libqemu and libvirt simultaneously which is certainly a key requirement of mine.

I think it's important that it's a dedicated monitor session though. It shouldn't just be injecting commands within an existing QMP session IMHO.

I think the opposite actually. If libvirt had two open monitor connections, one for normal use & one for injection, then its open to racey usage where 2 monitor commands may be issued concurrently & it is tricky to determine just which will be processed first, with all the scope for unexpected behaviour this entails. With a single monitor connection, libvirt's current locking model for the monitor ensures that QMP monitor commands are reliably serialized onto the wire, giving unambiguous behaviour. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On Fri, Apr 23, 2010 at 08:40:49AM -0500, Anthony Liguori wrote:

On 04/23/2010 05:28 AM, Daniel P. Berrange wrote:

...

On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote:

...

On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:

...

On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:

...

<domain type='kvm'> <name>myguest</name> ... <debug> <monitorpassthrough/> <commandline> <extra>qemu arguments</extra> <alter option="optname"> <rename>newname</rename> <match>REGEXP</match> <modify>foo=on</modify> <extra>-bar</extra> </alter> </commandline> </debug> </domain>

The concept of command line& monitor is something that is QEMU specific and thus is not suitable for the primary XML schema. IMHO, this needs to be done as a separate schema, linked in via an XML namespace. For example

<domain type='kvm' xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0"> <name>myguest</name> ... <qemu:commandline> <qemu:arg>-device</arg> <qemu:arg>lsi</arg> </qemu:commandline> </domain>

I think it's problematic to focus too much on command line arguments. We are not introducing new command line arguments to qemu for the most part that aren't usable in the config file.

Currently libvirt doesn't use any config file at all, so command line is the only possible option.

The config file is really just a structured command line argument so the two are really equivalent.

From an XML perspective, the advantage of using the config file format is that it's very well structured whereas the command line arguments aren't. It's the difference between:

<qemu:commandline> <arg>-drive</arg> <arg>file=foo.img,if=virtio,cache=off</arg> </qemu:commandline>

And:

<qemu:drive> <file>foo.img</file> <if>virtio</if> <cache>off</cache> </qemu:drive>

The later being much more friendly for things like XPath and XQuery.

...

We're going to evaluate switching to the config file at some point& so can easily add further XML options to allow direct setting of config file entries at that point.

I don't necessary think there's a super compelling reason for you to switch to pure config. Some things you'll have to do via -device and -set but it's still reasonable to keep using the command line. A key requirement of the config file effort was that everything that can be done via config file be doable via command line.

The main reason we're interested in the config is that it uses a better structured format, so we don't have such hairy escaping problems to deal with. Also, the command line is getting outrageously long these days with -device syntax.

...

...

With respect to injecting QMP commands directly, I think the proposed debug API is probably reasonable. We could build a libqemu that used that API as a transport which means that one could use libqemu and libvirt simultaneously which is certainly a key requirement of mine.

I think it's important that it's a dedicated monitor session though. It shouldn't just be injecting commands within an existing QMP session IMHO.

I think the opposite actually. If libvirt had two open monitor connections, one for normal use& one for injection, then its open to racey usage where 2 monitor commands may be issued concurrently& it is tricky to determine just which will be processed first, with all the scope for unexpected behaviour this entails.

Can you give an explicit example? The nature of this debug extension is such that I don't think 1 monitor really helps this problem...

Say libvirt is running a 'offline core dump' operation. This consists of us invoking stop migrate exec:cat > foo.dump cont I don't want other debug commands accidentally being issued in between these steps. These 3 commands are in essence considered transactional block. Internally our locking model ensures that no other APIs can be invoked in this monitor connection during this sequence. Thus if someone uses the libvirt debug API to issue a monitor command, thus command will be blocked until the last 'cont' command is issued here. This provides us a known command ordering in the logs.

...

With a single monitor connection, libvirt's current locking model for the monitor ensures that QMP monitor commands are reliably serialized onto the wire, giving unambiguous behaviour.

Except that libvirt doesn't know what the side effects of the debug commands are so it's intrinsically ambiguous :-)

This is a different ambiguity, about the semantic results of the commands, where as I'm refering to the execution order. If I look at a libvirt log file and see a set of JSON commands logged, I want to know that this ordering from the logs, was indeed the same as order in which qemu processed them. If you have two separate monitor connection you can't be sure of the order of execution. It is key for our bug troubleshooting that given a libvirt log file, we can replay the JSON commands again and get the same results. Two monitor connections is just increasing complexity of code without any tangible benefit. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On 04/23/2010 09:33 PM, Anthony Liguori wrote:

...

This is a different ambiguity, about the semantic results of the commands, where as I'm refering to the execution order. If I look at a libvirt log file and see a set of JSON commands logged, I want to know that this ordering from the logs, was indeed the same as order in which qemu processed them. If you have two separate monitor connection you can't be sure of the order of execution. It is key for our bug troubleshooting that given a libvirt log file, we can replay the JSON commands again and get the same results. Two monitor connections is just increasing complexity of code without any tangible benefit.

I think you're assuming direct access to the second monitor? I'm not suggesting that. I'm suggesting that libvirt is still the one submitting commands to the second monitor and that it submits those commands in lock step.

What about protocol extensions? For instance, pretend libvirt doesn't support async messages, what would it do when it receives one from the user's monitor? -- error compiling committee.c: too many arguments to function