[libvirt] Supporting hypervisor specific APIs in libvirt
Hi, I've mentioned this to a few folks already but I wanted to start a proper thread. We're struggling in qemu with usability and one area that concerns me is the disparity in features that are supported by qemu vs what's implemented in libvirt. This isn't necessarily libvirt's problem if it's mission is to provide a common hypervisor API that covers the most commonly used features. However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume. I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs. What's the feeling about this from the libvirt side of things? Is there interest in support hypervisor specific interfaces should we be looking to provide our own management interface for libvirt to consume? Regards, Anthony Liguori
On Mon, Mar 22, 2010 at 02:25:00PM -0500, Anthony Liguori wrote:
Hi,
I've mentioned this to a few folks already but I wanted to start a proper thread.
We're struggling in qemu with usability and one area that concerns me is the disparity in features that are supported by qemu vs what's implemented in libvirt.
This isn't necessarily libvirt's problem if it's mission is to provide a common hypervisor API that covers the most commonly used features.
That is more or less our current mission. If this mission leads to QEMU creating a non-libvirt based API & telling people to use that instead, then I'd say libvirt's mission needs to change to avoid that scenario ! I strongly believe that libvirt's strategy is good for application developers over the medium to long term. We need to figure out how to get rid of the short term pain from the feature timelag, rather than inventing a new library API for them to use.
However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume.
I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs.
Stepping back a bit first, there are the two core areas in which people can be limited by libvirt currently. 1. Monitor commands 2. Command line flags Ultimately, IIUC, you are suggesting we need to allow arbitrary passthrough for both of these in libvirt. At the libvirt level, we have 3 core requirements 1. The XML format is extend only (new elements allowed, or add attributes or children to existing elements) 2. The C library API is append only (new symbols only) 3. The RPC wire protocol is append only (maps 1-1 to the C API generally) The core question for us as libvirt developers is how we could support QEMU specific features that may change arbitrarily, without it impacting on our ability to maintain these 3 requirements for the non-hypervisor specific APIs. We don't ever want to be in a situation where a QEMU specific API will require us to change the soname of the main libvirt library, or introduce incompatible wire protocol changes. If we were to introduce QEMU specific APIs, we also need a way to easily remove those over time, as & when we have them available as generic APIs. At the C API level, this to me suggests that we'd want to introduce a separate libvirt-qemu.so library for the QEMU specific APIs. This library would not have the same requirements of fixed long term ABI that the main libvirt.so did. We'd add QEMU APIs to libvirt-qemu.so any time needed, but remove them when the equivalent functionality were in libvirt.so, and increment the soname of libvirt-qemu.so at that point. At the wire protocol level, the protocol allows us to support multiple versioned protocols in parallel over the same data stream. So again there, we could define a sub-protocol for QEMU specific features for which we don't provide the indefinite ABI compatability. Finally the XML format is "easy" - just have a versioned XML namespace for extra pieces, that's distinct from the default namespace, again without the permanent long term compatability guarentees. There are, however, some bits that are unlikely to work when QEMU is under libvirt. Specifically any of the device backends that use stdio (eg, -serial stdio, or the ncurses graphics), simply because all libvirt spawned VMs are fully daemonized & so stdio is /dev/null Other items are hard, but not entirely impossible to solve. eg, any use of the 'script=' arg for -net devices doesn't work, because libvirt clears all capabilities from the QEMU process so it'll be lacking CAP_NET_ADMIN which most TAP device setup scripts in fact need. Some parts of the C library/wire protocol here are related to another feature I'd like to introduce for libvirt, namely a administrative library. eg a API to configure and manage the libvirtd daemon itself on the fly. This could easily hook into the wire protocol, but live as a separate libvirt-daemon.so library API in similar way to what I suggest for QEMU specific API
What's the feeling about this from the libvirt side of things? Is there interest in support hypervisor specific interfaces should we be looking to provide our own management interface for libvirt to consume?
Adding yet another library in the stack isn't really going to solve the problem from the POV of libvirt users, but rather fork the community effort which I imagine we'd all rather avoid. Having to tell people to switch to a different library API to get access to a specific feature is a short term win, but with a significant long term cost/burden. This means we really do need to figure out how to better/fully support QEMU's features in libvirt, removing the feature timelag pain. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Hi,
Stepping back a bit first, there are the two core areas in which people can be limited by libvirt currently.
2. Command line flags
For me: This one, and monitor access. libvirt is very unfriendly to qemu hackers. There is no easy way to add command line switches. There is no easy way to get access to the monitor. I can get it done by pointing <emulator> to a wrapper script and mangle the qemu command line there. But this sucks big time. And it doesn't integrate with libvirt at all. When hacking qemu, especially when adding new command line options or monitor commands, I want to have a easy way to test this stuff. Or I just wanna able to type some 'info $foo' commands for debugging and trouble shooting purposes. libvirt makes it harder not easier to get the job done. Image you could ask libvirt to create an additional monitor and expose it like a serial console. virt-manager lists it as text console. Two mouse clicks open a new window (or tab) with a terminal emulator linked to the monitor. Wouldn't that be cool? Other issues I've trap into: -boot libvirt (or virt-manager?) supports only the very old single letter style. You can't specify '-boot order=cd,menu=on'. -enable-nested not available. serial console doesn't work for remote connections. cheers, Gerd
On 03/22/2010 04:33 PM, Gerd Hoffmann wrote:
Hi,
Stepping back a bit first, there are the two core areas in which people can be limited by libvirt currently.
2. Command line flags
For me: This one, and monitor access.
libvirt is very unfriendly to qemu hackers. There is no easy way to add command line switches. There is no easy way to get access to the monitor. I can get it done by pointing <emulator> to a wrapper script and mangle the qemu command line there. But this sucks big time. And it doesn't integrate with libvirt at all.
It's not just developers. As we're doing deployments of qemu/kvm, we keep running into the same problem. We realize that we need to use a feature of qemu/kvm that isn't modelled by libvirt today. I've gone as far as to temporarily pausing libvirtd, finding the pty fd from /proc/<pid>, and hijacking the monitor session temporarily. The problem is, it's not always easy to know what the most important features are.
When hacking qemu, especially when adding new command line options or monitor commands, I want to have a easy way to test this stuff. Or I just wanna able to type some 'info $foo' commands for debugging and trouble shooting purposes. libvirt makes it harder not easier to get the job done.
Image you could ask libvirt to create an additional monitor and expose it like a serial console. virt-manager lists it as text console. Two mouse clicks open a new window (or tab) with a terminal emulator linked to the monitor. Wouldn't that be cool?
Other issues I've trap into:
-boot libvirt (or virt-manager?) supports only the very old single letter style. You can't specify '-boot order=cd,menu=on'.
You can, you specify multiple <boot> options (but you can't touch things like menu=on). Regards, Anthony Liguori
-enable-nested not available.
serial console doesn't work for remote connections.
cheers, Gerd
On 03/22/10 22:53, Anthony Liguori wrote:
On 03/22/2010 04:33 PM, Gerd Hoffmann wrote:
libvirt is very unfriendly to qemu hackers. There is no easy way to add command line switches. There is no easy way to get access to the monitor. I can get it done by pointing <emulator> to a wrapper script and mangle the qemu command line there. But this sucks big time. And it doesn't integrate with libvirt at all.
It's not just developers. As we're doing deployments of qemu/kvm, we keep running into the same problem. We realize that we need to use a feature of qemu/kvm that isn't modelled by libvirt today. I've gone as far as to temporarily pausing libvirtd, finding the pty fd from /proc/<pid>, and hijacking the monitor session temporarily.
One problem I have found, and I am not sure how to fix this in this context. Sometimes when hacking on qemu, I want to try out a new qemu binary on an existing image, without replacing the system wide one and may want to pass new command line flags for testing those, plus have access to the monitor. What I do now is to look at the command line arguments of a guest using ps and try and mimic it, but due to the random magic ptys and other stuff, it's practically impossible to replicate a libvirt spawned qemu on the command line. I end up having a somewhat similar command line with everything removed that I cannot replicate. I find it a real problem that libvirt tries to wrap things to the point that an ordinary human cannot read, modify it's configuration or do a simple command line spawn to replicate it, but as I said, I am not sure how to solve the problem. Regards, Jes
On 03/23/10 09:54, Jes Sorensen wrote:
On 03/22/10 22:53, Anthony Liguori wrote:
On 03/22/2010 04:33 PM, Gerd Hoffmann wrote:
libvirt is very unfriendly to qemu hackers. There is no easy way to add command line switches. There is no easy way to get access to the monitor. I can get it done by pointing <emulator> to a wrapper script and mangle the qemu command line there. But this sucks big time. And it doesn't integrate with libvirt at all.
It's not just developers. As we're doing deployments of qemu/kvm, we keep running into the same problem. We realize that we need to use a feature of qemu/kvm that isn't modelled by libvirt today. I've gone as far as to temporarily pausing libvirtd, finding the pty fd from /proc/<pid>, and hijacking the monitor session temporarily.
One problem I have found, and I am not sure how to fix this in this context. Sometimes when hacking on qemu, I want to try out a new qemu binary on an existing image, without replacing the system wide one and may want to pass new command line flags for testing those, plus have access to the monitor.
Works with the wrapper script trick mentioned above. virsh edit $domain grep for <emulator> make it point to a wrapper script. My setup: [root@xenb ~]# virsh dumpxml fedora | grep emulator <emulator>/root/bin/qemu-wrapper</emulator> [root@xenb ~]# cat /root/bin/qemu-wrapper #!/bin/sh # distro qemu-kvm REAL_QEMU="/usr/bin/qemu-kvm" MORE_ARGS="-boot menu=on -cpu host -enable-nesting" # fresh build #REAL_QEMU="/home/kraxel/git/kvm/x86_64-softmmu/qemu-system-x86_64" #MORE_ARGS="-L /home/kraxel/git/kvm/pc-bios -boot menu=on" # go! case "$1" in "" | -h | -help | --help) # libvirt capability check exec $REAL_QEMU $1 ;; *) # run qemu with additional args exec $REAL_QEMU "$@" $MORE_ARGS ;; esac # should never ever arrive here echo "$0: exec $REAL_QEMU failed" >&2 exit 1 HTH, Gerd
On 03/23/10 11:25, Gerd Hoffmann wrote:
On 03/23/10 09:54, Jes Sorensen wrote:
One problem I have found, and I am not sure how to fix this in this context. Sometimes when hacking on qemu, I want to try out a new qemu binary on an existing image, without replacing the system wide one and may want to pass new command line flags for testing those, plus have access to the monitor.
Works with the wrapper script trick mentioned above.
virsh edit $domain grep for <emulator> make it point to a wrapper script.
My setup:
[root@xenb ~]# virsh dumpxml fedora | grep emulator <emulator>/root/bin/qemu-wrapper</emulator> [root@xenb ~]# cat /root/bin/qemu-wrapper
Ah right thanks! However, it's a hack to get around the real problem with libvirt. Not to mention that the output from virsh dumpxml is where you have to cover your eyes and try not getting sick while editing :( Having a normal config file in readable format where you could add regular command line options manually would make life so much easier. Cheers, Jes
On 03/23/10 11:31, Jes Sorensen wrote:
On 03/23/10 11:25, Gerd Hoffmann wrote:
[root@xenb ~]# virsh dumpxml fedora | grep emulator <emulator>/root/bin/qemu-wrapper</emulator> [root@xenb ~]# cat /root/bin/qemu-wrapper
Ah right thanks! However, it's a hack to get around the real problem with libvirt.
Sure. It is a hack, not a real solution. And it easily gets much more messy, for example in case you wanna have an additional monitor as unix:/var/tmp/mon/$vmname. cheers, Gerd
On 03/22/2010 05:33 PM, Gerd Hoffmann wrote:
Hi,
Stepping back a bit first, there are the two core areas in which people can be limited by libvirt currently.
2. Command line flags
For me: This one, and monitor access.
libvirt is very unfriendly to qemu hackers. There is no easy way to add command line switches. There is no easy way to get access to the monitor. I can get it done by pointing <emulator> to a wrapper script and mangle the qemu command line there. But this sucks big time. And it doesn't integrate with libvirt at all.
When hacking qemu, especially when adding new command line options or monitor commands, I want to have a easy way to test this stuff. Or I just wanna able to type some 'info $foo' commands for debugging and trouble shooting purposes. libvirt makes it harder not easier to get the job done.
Image you could ask libvirt to create an additional monitor and expose it like a serial console. virt-manager lists it as text console. Two mouse clicks open a new window (or tab) with a terminal emulator linked to the monitor. Wouldn't that be cool?
Other issues I've trap into:
-boot libvirt (or virt-manager?) supports only the very old single letter style. You can't specify '-boot order=cd,menu=on'.
Libvirt has supported multiple boot options for a while, it just wasn't in virt-manager. It's been upstream for a few weeks now though, and a new release is coming in a matter of days. I have a half implemented libvirt patch to allow setting boot menu, I guess it's time to dust it off :)
-enable-nested not available.
serial console doesn't work for remote connections.
Both of these have been requested a few times, so you aren't alone. - Cole
On 03/22/2010 03:10 PM, Daniel P. Berrange wrote:
This isn't necessarily libvirt's problem if it's mission is to provide a common hypervisor API that covers the most commonly used features.
That is more or less our current mission. If this mission leads to QEMU creating a non-libvirt based API& telling people to use that instead, then I'd say libvirt's mission needs to change to avoid that scenario ! I strongly believe that libvirt's strategy is good for application developers over the medium to long term. We need to figure out how to get rid of the short term pain from the feature timelag, rather than inventing a new library API for them to use.
Well that's certainly a good thing :-)
However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume.
I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs.
Stepping back a bit first, there are the two core areas in which people can be limited by libvirt currently.
1. Monitor commands 2. Command line flags
Ultimately, IIUC, you are suggesting we need to allow arbitrary passthrough for both of these in libvirt.
At the libvirt level, we have 3 core requirements
1. The XML format is extend only (new elements allowed, or add attributes or children to existing elements) 2. The C library API is append only (new symbols only) 3. The RPC wire protocol is append only (maps 1-1 to the C API generally)
We have a slightly different mentality within QEMU I think. Here's roughly how I'd characterize our guarantees. 1. For any two versions of QEMU, we try to guarantee that the same VM, as far as the guest sees it, can be created. 2. We tend to avoid changing command line syntax unless the syntax was previously undefined. 3. QMP supports enumeration and feature negotiation. This enables a client to discover which functions are supported. 4. We try to maintain monitor interfaces but provide no guarantees of compatibility.
The core question for us as libvirt developers is how we could support QEMU specific features that may change arbitrarily, without it impacting on our ability to maintain these 3 requirements for the non-hypervisor specific APIs.
We don't ever want to be in a situation where a QEMU specific API will require us to change the soname of the main libvirt library, or introduce incompatible wire protocol changes. If we were to introduce QEMU specific APIs, we also need a way to easily remove those over time, as& when we have them available as generic APIs.
At the C API level, this to me suggests that we'd want to introduce a separate libvirt-qemu.so library for the QEMU specific APIs. This library would not have the same requirements of fixed long term ABI that the main libvirt.so did. We'd add QEMU APIs to libvirt-qemu.so any time needed, but remove them when the equivalent functionality were in libvirt.so, and increment the soname of libvirt-qemu.so at that point.
How different is having a libvirt-qemu.so from having a libqemu.so that libvirt.so uses? Practically speaking, if libvirt-qemu.so uses a separate XML namespace, does the fact that we use a different config format matter since you can transform our config format to XML and vice versa? I think the problem is, if libvirt.so introduces a common API, removing it from libvirt-qemu.so is burdensome to an end-user. For someone designing a QEMU specific management application, why should they have to update their implementation to a common API that they'll never use?
At the wire protocol level, the protocol allows us to support multiple versioned protocols in parallel over the same data stream. So again there, we could define a sub-protocol for QEMU specific features for which we don't provide the indefinite ABI compatability.
Finally the XML format is "easy" - just have a versioned XML namespace for extra pieces, that's distinct from the default namespace, again without the permanent long term compatability guarentees.
There are, however, some bits that are unlikely to work when QEMU is under libvirt. Specifically any of the device backends that use stdio (eg, -serial stdio, or the ncurses graphics), simply because all libvirt spawned VMs are fully daemonized& so stdio is /dev/null
This seems fixable with //session.
Other items are hard, but not entirely impossible to solve. eg, any use of the 'script=' arg for -net devices doesn't work, because libvirt clears all capabilities from the QEMU process so it'll be lacking CAP_NET_ADMIN which most TAP device setup scripts in fact need.
Some parts of the C library/wire protocol here are related to another feature I'd like to introduce for libvirt, namely a administrative library. eg a API to configure and manage the libvirtd daemon itself on the fly. This could easily hook into the wire protocol, but live as a separate libvirt-daemon.so library API in similar way to what I suggest for QEMU specific API
What's the feeling about this from the libvirt side of things? Is there interest in support hypervisor specific interfaces should we be looking to provide our own management interface for libvirt to consume?
Adding yet another library in the stack isn't really going to solve the problem from the POV of libvirt users, but rather fork the community effort which I imagine we'd all rather avoid. Having to tell people to switch to a different library API to get access to a specific feature is a short term win, but with a significant long term cost/burden. This means we really do need to figure out how to better/fully support QEMU's features in libvirt, removing the feature timelag pain.
I think what we need to do is find a way to more tightly integrate the QEMU and libvirt communities in such a way that when a patch was submitted against QEMU adding a new feature, we could ask that that feature was implemented in libvirt. I see two ways to do this. One would be for libvirt to have a libvirt.so and libvirt-qemu.so. The QEMU community would have to be much more heavily involved in libvirt-qemu.so and it probably suggests that libvirt-qemu.so should follow our release cycle. libvirt would have to support using either libvirt.so or libvirt-qemu.so for it's users. The alternative would be for the QEMU community to produce a libqemu.so and for libvirt.so to consume libqemu.so. The libvirt community ought to be heavily engaged in the development of libqemu.so and certainly, shared maintainership would be appropriate. A user using libvirt.so should see guests created with either libqemu.so or libvirt.so although libqemu.so would provide weaker long term compatibility guarantees (but more features). I think both approaches are technically workable. The question is which is the most appealing to both communities and which requires the least development effort. At a high level, I think libqemu.so would require a heavy refactoring of the qemu libvirt driver. Guest enumeration would have to be handled by qemu and libvirt would need to support translating qemu's config format to an XML format. It would also need to translate the qemu XML format into it's own XML format relying on an xmlns to support the features that are not currently supported by the common hypervisor API. The advantage though would be that any qemu instance launched would be visible to libvirt and libvirt tooling. libvirt-qemu.so would require less change to the qemu driver in libvirt but would require some awkwardness in terms of translating a guest config to an XML config. Some of the concepts that qemu supports (like running a guest as non-privileged user using stdio) would be difficult to support. Regards, Anthony Liguori Regards, Anthony Liguori
Regards, Daniel
On 22.03.2010, at 22:49, Anthony Liguori wrote:
On 03/22/2010 03:10 PM, Daniel P. Berrange wrote:
What's the feeling about this from the libvirt side of things? Is there interest in support hypervisor specific interfaces should we be looking to provide our own management interface for libvirt to consume?
Adding yet another library in the stack isn't really going to solve the problem from the POV of libvirt users, but rather fork the community effort which I imagine we'd all rather avoid. Having to tell people to switch to a different library API to get access to a specific feature is a short term win, but with a significant long term cost/burden. This means we really do need to figure out how to better/fully support QEMU's features in libvirt, removing the feature timelag pain.
I think what we need to do is find a way to more tightly integrate the QEMU and libvirt communities in such a way that when a patch was submitted against QEMU adding a new feature, we could ask that that feature was implemented in libvirt. I see two ways to do this.
One would be for libvirt to have a libvirt.so and libvirt-qemu.so. The QEMU community would have to be much more heavily involved in libvirt-qemu.so and it probably suggests that libvirt-qemu.so should follow our release cycle. libvirt would have to support using either libvirt.so or libvirt-qemu.so for it's users.
The alternative would be for the QEMU community to produce a libqemu.so and for libvirt.so to consume libqemu.so. The libvirt community ought to be heavily engaged in the development of libqemu.so and certainly, shared maintainership would be appropriate. A user using libvirt.so should see guests created with either libqemu.so or libvirt.so although libqemu.so would provide weaker long term compatibility guarantees (but more features).
I don't see why we shouldn't be able to automatically generate libqemu.so. We have the *.hx files that describe the syntax of parameters plus list all available options / commands. I'm not sure how exactly QMP works, but having a generic QMP command to list all available options would be handy too. By then we could automate most of the library, making the glueing really easy. If libvirt doesn't properly link against libqemu anymore we also know why: The ABI changed. All that's needed then is a common Qemu object that libvirt users can get access to as well. That object is the magic key to all libqemu functions. If users need functionality not exposed in libvirt, they can then use libqemu calls directly. If they don't care about all the awesomeness of libvirt and don't want to be hypervisor agnostic, they can stick to libqemu completely. Alex
Alexander Graf wrote:
I don't see why we shouldn't be able to automatically generate libqemu.so. We have the *.hx files that describe the syntax of parameters plus list all available options / commands. I'm not sure how exactly QMP works, but having a generic QMP command to list all available options would be handy too.
By then we could automate most of the library, making the glueing really easy. If libvirt doesn't properly link against libqemu anymore we also know why: The ABI changed.
I'm thinking most potential uses of the binary API, other than C programmers, would be happier with a D-Bus version generated from those same *.hx files. Because then it's easy to call the API from Python, Perl, even shell, etc. Whereas libqemu.so would be relatively difficult to use from those languages. (Aside: I don't particularly like D-Bus. But it does seem to work for this sort of thing.) -- Jamie
On 03/23/2010 06:25 PM, Jamie Lokier wrote:
Alexander Graf wrote:
I don't see why we shouldn't be able to automatically generate libqemu.so. We have the *.hx files that describe the syntax of parameters plus list all available options / commands. I'm not sure how exactly QMP works, but having a generic QMP command to list all available options would be handy too.
By then we could automate most of the library, making the glueing really easy. If libvirt doesn't properly link against libqemu anymore we also know why: The ABI changed.
I'm thinking most potential uses of the binary API, other than C programmers, would be happier with a D-Bus version generated from those same *.hx files. Because then it's easy to call the API from Python, Perl, even shell, etc. Whereas libqemu.so would be relatively difficult to use from those languages.
My thinking with respect to libqemu.so is that it should be mostly autogenerated. QMP supports introspection, we should be able to generate an idl description of QMP via introspection and then build all of the function stubs from that idl. Then there is no opportunity for libqemu to be out of date. All we really need to write for libqemu is some core bits to deal with transport specific issues.
(Aside: I don't particularly like D-Bus. But it does seem to work for this sort of thing.)
I don't think d-bus is a good fit as a core qemu service. It's not commonly used on other platform and it introduces quite a bit of overhead for non-Unix platforms. But that certainly doesn't mean that a d-bus service implemented on top of libqemu (even autogenerated from our IDL) would be a bad project. Regards, Anthony Liguori
-- Jamie
Anthony Liguori <anthony@codemonkey.ws> writes:
On 03/23/2010 06:25 PM, Jamie Lokier wrote:
Alexander Graf wrote:
I don't see why we shouldn't be able to automatically generate libqemu.so. We have the *.hx files that describe the syntax of parameters plus list all available options / commands. I'm not sure how exactly QMP works, but having a generic QMP command to list all available options would be handy too.
Yes, the plan is to have QMP describe itself. Needs work.
By then we could automate most of the library, making the glueing really easy. If libvirt doesn't properly link against libqemu anymore we also know why: The ABI changed.
I can't quite see what such a libqemu would buy us compared to straight QMP. Talking QMP should be easy, provided you got a suitable JSON library. Generating a libqemu.so for (a particular version of) QMP may make talking (that version of) QMP slightly easier in C, by turning a simple text network protocol into a C API. But it's not without drawbacks. The text protocol is designed to be evolvable in backward-compatible ways. For instance, we can new add commands, new optional arguments, and so forth. But you can't add new optional arguments to C functions without changing the API. You can change the function and bump the soname, or you can deprecate the function and add a new one, or you can bypass static typing, e.g. by passing arguments in a dictionary. In the latter case, why not put the command in the dictionary as well, and cut the number of functions from N to 1. Ensured consistency of libqmp and QEMU sounds nice. But it's consistent with a *local* version of QEMU. QMP is a *network* protocol. If your app talks QMP straight, it can handle any remote version it knows all by itself. If you interpose libqmp, you add a dependency: you need a sufficiently current *local* QEMU/libqmp.
I'm thinking most potential uses of the binary API, other than C programmers, would be happier with a D-Bus version generated from those same *.hx files. Because then it's easy to call the API from Python, Perl, even shell, etc. Whereas libqemu.so would be relatively difficult to use from those languages.
I suspect importing a foreign libqmp.so C API into a non-C language is no easier than using the language's JSON facilities and talk QMP straight, and less flexible.
My thinking with respect to libqemu.so is that it should be mostly autogenerated.
QMP supports introspection, we should be able to generate an idl description of QMP via introspection and then build all of the function stubs from that idl. Then there is no opportunity for libqemu to be out of date.
All we really need to write for libqemu is some core bits to deal with transport specific issues.
I can't quite see the utility of that. [D-Bus snipped...]
On Mon, Mar 22, 2010 at 04:49:21PM -0500, Anthony Liguori wrote:
On 03/22/2010 03:10 PM, Daniel P. Berrange wrote:
This isn't necessarily libvirt's problem if it's mission is to provide a common hypervisor API that covers the most commonly used features.
That is more or less our current mission. If this mission leads to QEMU creating a non-libvirt based API& telling people to use that instead, then I'd say libvirt's mission needs to change to avoid that scenario ! I strongly believe that libvirt's strategy is good for application developers over the medium to long term. We need to figure out how to get rid of the short term pain from the feature timelag, rather than inventing a new library API for them to use.
Well that's certainly a good thing :-)
However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume.
I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs.
Stepping back a bit first, there are the two core areas in which people can be limited by libvirt currently.
1. Monitor commands 2. Command line flags
Ultimately, IIUC, you are suggesting we need to allow arbitrary passthrough for both of these in libvirt.
At the libvirt level, we have 3 core requirements
1. The XML format is extend only (new elements allowed, or add attributes or children to existing elements) 2. The C library API is append only (new symbols only) 3. The RPC wire protocol is append only (maps 1-1 to the C API generally)
We have a slightly different mentality within QEMU I think. Here's roughly how I'd characterize our guarantees.
1. For any two versions of QEMU, we try to guarantee that the same VM, as far as the guest sees it, can be created. 2. We tend to avoid changing command line syntax unless the syntax was previously undefined. 3. QMP supports enumeration and feature negotiation. This enables a client to discover which functions are supported. 4. We try to maintain monitor interfaces but provide no guarantees of compatibility.
The core question for us as libvirt developers is how we could support QEMU specific features that may change arbitrarily, without it impacting on our ability to maintain these 3 requirements for the non-hypervisor specific APIs.
We don't ever want to be in a situation where a QEMU specific API will require us to change the soname of the main libvirt library, or introduce incompatible wire protocol changes. If we were to introduce QEMU specific APIs, we also need a way to easily remove those over time, as& when we have them available as generic APIs.
At the C API level, this to me suggests that we'd want to introduce a separate libvirt-qemu.so library for the QEMU specific APIs. This library would not have the same requirements of fixed long term ABI that the main libvirt.so did. We'd add QEMU APIs to libvirt-qemu.so any time needed, but remove them when the equivalent functionality were in libvirt.so, and increment the soname of libvirt-qemu.so at that point.
How different is having a libvirt-qemu.so from having a libqemu.so that libvirt.so uses?
Practically speaking, if libvirt-qemu.so uses a separate XML namespace, does the fact that we use a different config format matter since you can transform our config format to XML and vice versa?
If an application used libqemu.so directly, they would be excluding themselves from much of the libvirt ecosystem which would otherwise be beneficial to their needs. For example, with libvirt-qemu.so, you could still use libvirt's secure remote API access, all the authentication & authorization capabilities on the API. It could also be intergrated into the other libvirt language bindings, and mapping layers such as libvirt-CIM, libvirt-qpid, etc. Applications may still also want the benefit of the libvirt hypervisor-agnostic APIs, for example, virt-manager wants to use libvirt.so primary so it can support QEMU, Xen, VMWare etc, but it might also want to access qemu specific features via libvirt-qemu.so It could not use libqemu.so because it would not be accessible via the libvirt remote RPC layer.
I think the problem is, if libvirt.so introduces a common API, removing it from libvirt-qemu.so is burdensome to an end-user. For someone designing a QEMU specific management application, why should they have to update their implementation to a common API that they'll never use?
I think we'd have to have a formal deprecation cycle, to allow overlap between adding a feature to libvirt.so & removing it from libvirt-qemu.so Even if they are building a QEMU specific mgmt application, there is still value in switching to a libvirt.so based API, because with that we are in a stronger position to guarentee their app will continue working with future QEMU that may change its command line or monitor syntax. For example, consider if a libvirt already had support for passing qemu specific args through to QEMU, and that app were using '-net' for some host style networking option libvirt.so didn't initially support. Now QEMU introduced '-netdev' as an improved option for networking. If the QEMU specific mgmt app had switched to the generic libvirt API once it was available, they would have got the benefit of the new improved -netdev option automatically. This is even more compelling in cases where QEMU has actually removed the existing argument.
What's the feeling about this from the libvirt side of things? Is there interest in support hypervisor specific interfaces should we be looking to provide our own management interface for libvirt to consume?
Adding yet another library in the stack isn't really going to solve the problem from the POV of libvirt users, but rather fork the community effort which I imagine we'd all rather avoid. Having to tell people to switch to a different library API to get access to a specific feature is a short term win, but with a significant long term cost/burden. This means we really do need to figure out how to better/fully support QEMU's features in libvirt, removing the feature timelag pain.
I think what we need to do is find a way to more tightly integrate the QEMU and libvirt communities in such a way that when a patch was submitted against QEMU adding a new feature, we could ask that that feature was implemented in libvirt. I see two ways to do this.
One would be for libvirt to have a libvirt.so and libvirt-qemu.so. The QEMU community would have to be much more heavily involved in libvirt-qemu.so and it probably suggests that libvirt-qemu.so should follow our release cycle. libvirt would have to support using either libvirt.so or libvirt-qemu.so for it's users.
I'm not anticipating an either/or scenario for this. Apps would primarily still use libvirt.so for everything. They would only use libvirt-qemu.so to access indiviaul QEMU specific APIs they might need. In other words, it augments the functionality of the base libvirt library.
The alternative would be for the QEMU community to produce a libqemu.so and for libvirt.so to consume libqemu.so. The libvirt community ought to be heavily engaged in the development of libqemu.so and certainly, shared maintainership would be appropriate. A user using libvirt.so should see guests created with either libqemu.so or libvirt.so although libqemu.so would provide weaker long term compatibility guarantees (but more features).
I think both approaches are technically workable. The question is which is the most appealing to both communities and which requires the least development effort.
I don't neccessarily think that the approaches are mutually exclusive either, because in some ways they are addressing different problem spaces. Even if there was a libqemu.so that libvirt used, there would still be demand for a libvirt-qemu.so, because apps using libvirt for mgmt do not want to discard use of the entire libvirt API just to access 1 extra QEMU specific feature. A libqemu.so doesn't solve that problem, its just an invisible internal impl detail for libvirt as far as end users are concerned.
At a high level, I think libqemu.so would require a heavy refactoring of the qemu libvirt driver. Guest enumeration would have to be handled by qemu and libvirt would need to support translating qemu's config format to an XML format. It would also need to translate the qemu XML format into it's own XML format relying on an xmlns to support the features that are not currently supported by the common hypervisor API.
Both of those are minor implementation details ;-)
The advantage though would be that any qemu instance launched would be visible to libvirt and libvirt tooling.
libvirt-qemu.so would require less change to the qemu driver in libvirt but would require some awkwardness in terms of translating a guest config to an XML config. Some of the concepts that qemu supports (like running a guest as non-privileged user using stdio) would be difficult to support.
Translating from libvirt XML to a QEMU specific config format is really not a burden. This is a core task for any libvirt hypervisor driver. It just happens that the current "config format" is a list of ARGV. Within reason, any other config format is just as managable. It only becomes hard when a hypervisor's config format arbitrarily changes semantics with each new release :-( Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Mon, Mar 22, 2010 at 04:49:21PM -0500, Anthony Liguori wrote:
On 03/22/2010 03:10 PM, Daniel P. Berrange wrote:
This isn't necessarily libvirt's problem if it's mission is to provide a common hypervisor API that covers the most commonly used features.
That is more or less our current mission. If this mission leads to QEMU creating a non-libvirt based API& telling people to use that instead, then I'd say libvirt's mission needs to change to avoid that scenario ! I strongly believe that libvirt's strategy is good for application developers over the medium to long term. We need to figure out how to get rid of the short term pain from the feature timelag, rather than inventing a new library API for them to use.
Well that's certainly a good thing :-)
However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume.
I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs.
Stepping back a bit first, there are the two core areas in which people can be limited by libvirt currently.
1. Monitor commands 2. Command line flags
Ultimately, IIUC, you are suggesting we need to allow arbitrary passthrough for both of these in libvirt.
At the libvirt level, we have 3 core requirements
1. The XML format is extend only (new elements allowed, or add attributes or children to existing elements) 2. The C library API is append only (new symbols only) 3. The RPC wire protocol is append only (maps 1-1 to the C API generally)
We have a slightly different mentality within QEMU I think. Here's roughly how I'd characterize our guarantees.
1. For any two versions of QEMU, we try to guarantee that the same VM, as far as the guest sees it, can be created. 2. We tend to avoid changing command line syntax unless the syntax was previously undefined. 3. QMP supports enumeration and feature negotiation. This enables a client to discover which functions are supported. 4. We try to maintain monitor interfaces but provide no guarantees of compatibility.
Points 2 & 4 make it very hard for libvirt to use any library API that QEMU might expose. We need to support multiple concurrently running versions of QEMU on a host, to cope with the package upgrade scenario & adhoc testing of new versions. If a libqemu.so for talking to QEMU changed a monitor interface & didn't have backwards compatability for older QEMU version, then it is not something we could use, because any particular libvirt build would be tied to only being able to talk to the specific QEMU version. Currently we internally deal with changes in syntax detecting which format/protocol we need to use at runtime and need to maintain that ability. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Mon, Mar 22, 2010 at 02:25:00PM -0500, Anthony Liguori wrote:
Hi,
I've mentioned this to a few folks already but I wanted to start a proper thread.
We're struggling in qemu with usability and one area that concerns me is the disparity in features that are supported by qemu vs what's implemented in libvirt.
This isn't necessarily libvirt's problem if it's mission is to provide a common hypervisor API that covers the most commonly used features.
However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume.
I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs.
The second thing I forgot to mention in my previous reply, is that we also need to get a much better idea of just which QEMU features missing in libvirt. Even if we provide a generic mechansim for setting QEMU specific features, we still need visibility into what needs to be added to the main generic libvirt XML / API format. I think the qdev & QMP work will be very useful in this respect, since both are pretty much self-describing. All that is missing from QEMU is a way to query/extra the qdev/QMP metadata from the QEMU binary in an automated fashion. eg, '-device qdev' can list all devices, but following on from that we need to query the properties known against each device type. I think it'd also be desirable to have a machine readable '-help' output format, so we can more reliably detect what args are available, since even with qdev + -device, we're still adding more command line args to QEMU over time like -netdev, -blockdev, etc. If we can easily get these details of qdev properties, QMP commands/args and ARGV from QEMU, then we can reliably identify just what is missing from libvirt & use that to guide development of generic APIs for the missing features. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Anthony Liguori <anthony@codemonkey.ws> wrote:
Hi,
I've mentioned this to a few folks already but I wanted to start a proper thread.
We're struggling in qemu with usability and one area that concerns me is the disparity in features that are supported by qemu vs what's implemented in libvirt.
This isn't necessarily libvirt's problem if it's mission is to provide a common hypervisor API that covers the most commonly used features.
However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume.
I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs.
What's the feeling about this from the libvirt side of things? Is there interest in support hypervisor specific interfaces should we be looking to provide our own management interface for libvirt to consume?
My problems with libvirt are: - I can't "hack" the qemu command line. It would be great to be able to edit the command line that libvirt is about to launch (yes, this is useful for developers, but it is an easy hack). - monitor: I need a way to get to the monitor when going through libvirt, in the past you couldn't allow this, but now it looks possible. - changing the emulator: It is normal for me to be using several qemu binaries for testing, changing it with libvirt is just a mess. Now, why I want to use libvirt in the 1st place? - libvirt is good at remembering details like mac addresses, disks positions, etc. Compare virsh start foo versus a 300+ character long qemu command line. - virt-viewer: I want to be able to switch consoles, period. - networking: man, setting networking is a mess, libvirt just does it for you. Now, to do suggestions. What would make it for me: - being able to use the qemu command line -> maximum flexibility - if that is too complicated for any reason, being able to change the <emulator>...</emulator> would be good enough the mayority of the time. What I normally need is: use a different emulator or add some additional flag to qemu. Just my 2 cents Juan.
- networking: man, setting networking is a mess, libvirt just does it for you.
+1 Even when not using libvirt for a reason or another I usually hook my virtual machines into virbr0 (libvirt default network). cheers, Gerd
Gerd Hoffmann <kraxel@redhat.com> wrote:
- networking: man, setting networking is a mess, libvirt just does it for you.
+1
Even when not using libvirt for a reason or another I usually hook my virtual machines into virbr0 (libvirt default network).
This is a war for another day :-) I have that very same setup on my laptop. But I already use dnsmasq for other reasons -> no way to share dnsmasq with libvirt, libvirt want to use a different one -> have to configure two things, especially if I want specific names (in my dnsmasq setup already) for some of my guests. Later, Juan.
On Tue, Mar 23, 2010 at 11:50:57AM +0100, Juan Quintela wrote:
Gerd Hoffmann <kraxel@redhat.com> wrote:
- networking: man, setting networking is a mess, libvirt just does it for you.
+1
Even when not using libvirt for a reason or another I usually hook my virtual machines into virbr0 (libvirt default network).
This is a war for another day :-)
I have that very same setup on my laptop. But I already use dnsmasq for other reasons -> no way to share dnsmasq with libvirt, libvirt want to use a different one -> have to configure two things, especially if I want specific names (in my dnsmasq setup already) for some of my guests.
FYI, the dnsmasq maintainer has actually volunteered todo some enhancements to dnsmasq to allow a prexisting instance of dnsmasq to play nicely with libvirt's needs. This should pretty much solve that problem Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
"Daniel P. Berrange" <berrange@redhat.com> wrote:
On Tue, Mar 23, 2010 at 11:50:57AM +0100, Juan Quintela wrote:
Gerd Hoffmann <kraxel@redhat.com> wrote:
- networking: man, setting networking is a mess, libvirt just does it for you.
+1
Even when not using libvirt for a reason or another I usually hook my virtual machines into virbr0 (libvirt default network).
This is a war for another day :-)
I have that very same setup on my laptop. But I already use dnsmasq for other reasons -> no way to share dnsmasq with libvirt, libvirt want to use a different one -> have to configure two things, especially if I want specific names (in my dnsmasq setup already) for some of my guests.
FYI, the dnsmasq maintainer has actually volunteered todo some enhancements to dnsmasq to allow a prexisting instance of dnsmasq to play nicely with libvirt's needs. This should pretty much solve that problem
Great!!!!! /me reserves beer for him. Once there, making it to play well with selinux + dbus will make it just perfect :-) (I haven't checked if this worked in the last year, so ...) Later, Juan.
Gerd Hoffmann wrote:
- networking: man, setting networking is a mess, libvirt just does it for you.
+1
Even when not using libvirt for a reason or another I usually hook my virtual machines into virbr0 (libvirt default network).
I had the opposite problem. Needed to use multiple bridges and have some VMs behind NAT without a bridge (private IPs), and some using separately firewalled bridges (needed to behave like real attached hardware with their original MACs, but be firewalled). I couldn't see how to do it easily with libvirt, so used qemu directly. -- Jamie
On 03/24/10 00:13, Jamie Lokier wrote:
Gerd Hoffmann wrote:
- networking: man, setting networking is a mess, libvirt just does it for you.
+1
Even when not using libvirt for a reason or another I usually hook my virtual machines into virbr0 (libvirt default network).
I had the opposite problem. Needed to use multiple bridges and have some VMs behind NAT without a bridge (private IPs), and some using separately firewalled bridges (needed to behave like real attached hardware with their original MACs, but be firewalled).
No problem in theory. libvirt should detect existing bridges and allow you to attach virtual machines to them. So you can setup bridges and firewalling for them using usual distro tools and use them for virtual machines. In practice I've seen this not working correctly in the past, i.e. my br0 didn't pop up in the virt-manager nic setup page. cheers, Gerd
On 03/24/2010 03:59 AM, Gerd Hoffmann wrote:
On 03/24/10 00:13, Jamie Lokier wrote:
Gerd Hoffmann wrote:
- networking: man, setting networking is a mess, libvirt just does it for you.
+1
Even when not using libvirt for a reason or another I usually hook my virtual machines into virbr0 (libvirt default network).
I had the opposite problem. Needed to use multiple bridges and have some VMs behind NAT without a bridge (private IPs), and some using separately firewalled bridges (needed to behave like real attached hardware with their original MACs, but be firewalled).
No problem in theory. libvirt should detect existing bridges and allow you to attach virtual machines to them. So you can setup bridges and firewalling for them using usual distro tools and use them for virtual machines.
In practice I've seen this not working correctly in the past, i.e. my br0 didn't pop up in the virt-manager nic setup page.
Please file a bug: virt-manager has had bridge detection for years, so something must be going wrong. In f13 we will use netcf for this, so even bridge enumeration on remote hosts should work. Additionally I recently made a change upstream to allow users to manually enter a bridge name, since netcf isn't supported for all distros yet. - Cole
In practice I've seen this not working correctly in the past, i.e. my ^^^^^^^^^^^ br0 didn't pop up in the virt-manager nic setup page.
Please file a bug: virt-manager has had bridge detection for years, so something must be going wrong.
Works for me now ;)
In f13 we will use netcf for this, so even bridge enumeration on remote hosts should work.
Yes, remote connections. F12 + virt-preview works. cheers, Gerd
Juan Quintela wrote:
- monitor: I need a way to get to the monitor when going through libvirt, in the past you couldn't allow this, but now it looks possible.
Now you can just start another monitor connection to qemu :-) Previously I've used a multiplexing script which accepts multiple monitor connections, and passes the commands to the real connection. It's not ideal because a slow command blocks any others, but it basically works.
- changing the emulator: It is normal for me to be using several qemu binaries for testing, changing it with libvirt is just a mess.
Same here, except not just for testing: I need to use different qemu binaries for production use, because newer ones don't work with some VM images that work on older ones.
- virt-viewer: I want to be able to switch consoles, period.
A decent VNC client gets close to this. If the qemus advertised themselves, some VNC clients would show a list of them automatically. As it is, I use Gnome's VNC client with a bookmark for each VM, which works quite well. It even has tabs :-)
- networking: man, setting networking is a mess, libvirt just does it for you.
networking is often a mess, and what libvirt does isn't always what you want, even if it often is. I need the option to set up networking separately and have libvirt use what it's given, otherwise I cannot use it. -- Jamie
Juan Quintela <quintela@redhat.com> writes:
- networking: man, setting networking is a mess, libvirt just does it for you.
Agreed it's messy, but isn't this something that the standard qemu command line tool could potentially do better by itself? I don't see why you need a wrapper for that. -Andi -- ak@linux.intel.com -- Speaking for myself only.
Andi Kleen <andi@firstfloor.org> wrote:
Juan Quintela <quintela@redhat.com> writes:
- networking: man, setting networking is a mess, libvirt just does it for you.
Agreed it's messy, but isn't this something that the standard qemu command line tool could potentially do better by itself? I don't see why you need a wrapper for that.
In my case, basically it is MAC addresses. I have dhcp setup, and it always give the same IP to the same MAC. But you have to remember to type the MAC addresses. This is the typical command line that virsh start launch for me: /usr/libexec/qemu-kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name f12X-64 -uuid 1fbe73a6-f519-e848-03bd-6636f765d143 -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/f12X-64.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot c -drive file=/mnt/kvm/images/f12X-64.img,if=none,id=drive-virtio-disk0,boot=on,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -device virtio-net-pci,vlan=0,id=net0,mac=54:52:00:44:72:e6,bus=pci.0,addr=0x5 -net tap,fd=18,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k es -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
From parts:
/usr/libexec/qemu-kvm -S I don't want that -M pc-0.12 I don't care. -enable-kvm I _want_ :) -m 1024 Also god idea -smp 2,sockets=2,cores=1,threads=1 by hand it is always "-smp 2" -name f12X-64 -uuid 1fbe73a6-f519-e848-03bd-6636f765d143 don't care -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/f12X-64.monitor,server,nowait -mon chardev=monitor,mode=readline this is simplified as: -monitor stdio when I launch it by hand. -rtc base=utc -boot c don't care -drive file=/mnt/kvm/images/f12X-64.img,if=none,id=drive-virtio-disk0,boot=on,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 this is _wow_, I only want to put the disk image path and convince it to use virtio driver -device virtio-net-pci,vlan=0,id=net0,mac=54:52:00:44:72:e6,bus=pci.0,addr=0x5 -net tap,fd=18,vlan=0,name=hostnet0 this always have to be changed. s/fd=18/script=/etc/kvm-ifup/ and then I normally found that I want "downscript=" to avoid the warning at exit time. If I don't put a mac address, qemu command line works well, but as I normally also use vnc I have to: - launch qemu - kill it, relaunch with -vnc :0 instead of -vnc 127.0.0.1:0 - re-launch qemu - connect to vnc - check what address the dhcp server was giving to it this time - I can ssh to the client now with libvirt handling the command line, I just ssh to the same dhcp address that was given the previous time/day/... -chardev pty,id=serial0 -device isa-serial,chardev=serial0 I only use serial from time to time, and using -serial tcp:0,server,nowait (or whatever is the sintax is easier by hand) -usb -device usb-tablet,id=input0 usb tablet is mandatory, just in case the guest is able to _not_ grab the mouse. -vnc 127.0.0.1:0 Allways wrong in my case, because I want to run the vnc client in a different machine. a way to convince virt-viewer to connect to a qemu launched by hand, or a way to convince libvirt to let me edit the command line will be great. -k es -vga cirrus this get right by default. -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 I normally don't use balloon. Notice for the "normally I don't care bits", that at the end I always care. Why? because then somebody arrives and told me that sound don't work, and I have to edit the config file, and add sound option. add a sound option to the command line of qemu is not too complicate. The other big problem for me are snapshots, I have to remember _exactly_ what was the qemu command line with which I saved the snapshot. Guess what, I normally don't remember and end: - launching old qemu - save a new snapshot - test with the new qemu and new snapshot (because now I have the command line that I launched 5 mins before). Just in case it helps. Later, Juan.
On Mon, Mar 22, 2010 at 02:25:00PM -0500, Anthony Liguori wrote:
Hi,
Hi Anthony,
I've mentioned this to a few folks already but I wanted to start a proper thread.
We're struggling in qemu with usability and one area that concerns me is the disparity in features that are supported by qemu vs what's implemented in libvirt.
If you could come up with a list, then I would have an easier job answering, honnestly I have the feeling we spent the last 6 months filling that gap in a really fast way.
This isn't necessarily libvirt's problem if it's mission is to provide a common hypervisor API that covers the most commonly used features.
"most commonly used" is not the point of libvirt
However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume.
I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs.
But one point of libvirt is that once an API is there we don't break it. I think there is a serious divergence of approach there, instanciating API stating 'we are gonna deprecate them sooner or later' tell the application developper 'my time is more important than yours' and not really something I like to carry to the API users. The main goal of libvirt remains to provide APIs needed to unify the development of the virtualization layers. Having APIs which makes sense only for one or 2 virtualization engines is not a problem in itself, it just raises questions about the actual semantic of that API. If that semantic is sound, then I see no reason to not add it, really and we actually often do.
What's the feeling about this from the libvirt side of things? Is there interest in support hypervisor specific interfaces should we be looking to provide our own management interface for libvirt to consume?
The real question is what do you actually want to build. Most of the feedback I have seen in this thread so far are mostly request to be able to hack on a qemu instance launched via libvirt. That looks a lot more an usability problem from the KVM/QEmu developpers than a problem related to building long term applications using libvirt stack and specific QEmu feature. As Dan answered the problem seems to be the short term, when features are being developped or tested for QEmu/KVM, I think once the feature is there we do try to provide support, and more general integration in the OS framework. I actually think we are fairly fast to roll in support for most feature, snapshot being one notable exception. Maybe more hacking support is needed, i.e. some way to access the monitor or add arbitrary qemu command line in a non-supported way, the main goal being to ease the life of developpers of QEmu/KVM, but that non-supported status need to be made clear. That's runtime facilities, and somehow I feel there is a point for this, though clearly that would for example be disabled for 'enterprise' builds. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
On 03/23/2010 09:51 AM, Daniel Veillard wrote:
On Mon, Mar 22, 2010 at 02:25:00PM -0500, Anthony Liguori wrote:
Hi,
Hi Anthony,
I've mentioned this to a few folks already but I wanted to start a proper thread.
We're struggling in qemu with usability and one area that concerns me is the disparity in features that are supported by qemu vs what's implemented in libvirt.
If you could come up with a list, then I would have an easier job answering, honnestly I have the feeling we spent the last 6 months filling that gap in a really fast way.
qemu-doc.texi is a list of most of the command line features we support. The help output of the monitor shows what we support in that interface. It doesn't take a lot to read through it and see the things not supported by libvirt. libvirt supports a relatively small amount of our overall features (although a good chunk of the most common set).
However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume.
I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs.
But one point of libvirt is that once an API is there we don't break it.
I think there is a serious divergence of approach there, instanciating API stating 'we are gonna deprecate them sooner or later' tell the application developper 'my time is more important than yours' and not really something I like to carry to the API users. The main goal of libvirt remains to provide APIs needed to unify the development of the virtualization layers. Having APIs which makes sense only for one or 2 virtualization engines is not a problem in itself, it just raises questions about the actual semantic of that API. If that semantic is sound, then I see no reason to not add it, really and we actually often do.
Yeah, but the problem we're facing is, I want there to be an API added to the management layer as part of the feature commit in qemu. If there has to be a discussion and decisions about how to model the API, it's not going to be successful. Supporting legacy APIs forever is not a viable option for a project like qemu. Things evolve quickly and we need a mechanism to deprecate APIs over time.
What's the feeling about this from the libvirt side of things? Is there interest in support hypervisor specific interfaces should we be looking to provide our own management interface for libvirt to consume?
The real question is what do you actually want to build.
Any management application really. Even with something like virt-manager, there's a ton of useful features that qemu supports (like migration status reporting) that libvirt doesn't support.
Most of the feedback I have seen in this thread so far are mostly request to be able to hack on a qemu instance launched via libvirt.
It's not about the "hacker" use-case. It's about making sure that we've got 100% feature coverage in our management API. All of the management tools that focus on KVM have had this problem that I am aware of. We need to come up with a way that we can very easily plumb new qemu functions through the management interface. Regards, Anthony Liguori
On 03/22/2010 09:25 PM, Anthony Liguori wrote:
Hi,
I've mentioned this to a few folks already but I wanted to start a proper thread.
We're struggling in qemu with usability and one area that concerns me is the disparity in features that are supported by qemu vs what's implemented in libvirt.
This isn't necessarily libvirt's problem if it's mission is to provide a common hypervisor API that covers the most commonly used features.
However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume.
I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs.
What's the feeling about this from the libvirt side of things? Is there interest in support hypervisor specific interfaces should we be looking to provide our own management interface for libvirt to consume?
One option is to expose a qmp connection to the client. Of course that introduces a consistency problem (libvirt plugs in a card, user plugs it own, libvirt is confused). If the user promises to behave, it can work for stuff that's 100% orthogonal to libvirt. One problem is that this is libvirt version specific. For example, libvirt x doesn't support spice so we control that thorough qmp. But libvirt x+1 does support spice and now it gets confused about all the spice messages. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.
On 03/23/2010 12:57 PM, Avi Kivity wrote:
On 03/22/2010 09:25 PM, Anthony Liguori wrote:
Hi,
I've mentioned this to a few folks already but I wanted to start a proper thread.
We're struggling in qemu with usability and one area that concerns me is the disparity in features that are supported by qemu vs what's implemented in libvirt.
This isn't necessarily libvirt's problem if it's mission is to provide a common hypervisor API that covers the most commonly used features.
However, for qemu, we need an API that covers all of our features that people can develop against. The ultimate question we need to figure out is, should we encourage our users to always use libvirt or should we build our own API for people (and libvirt) to consume.
I don't think it's necessarily a big technical challenge for libvirt to support qemu more completely. I think it amounts to introducing a series of virQemuXXXX APIs that implement qemu specific functions. Over time, qemu specific APIs can be deprecated in favour of more generic virDomain APIs.
What's the feeling about this from the libvirt side of things? Is there interest in support hypervisor specific interfaces should we be looking to provide our own management interface for libvirt to consume?
One option is to expose a qmp connection to the client. Of course that introduces a consistency problem (libvirt plugs in a card, user plugs it own, libvirt is confused). If the user promises to behave, it can work for stuff that's 100% orthogonal to libvirt.
With GTK/GDK/Cairo/etc, the expectation is that users behave. For things like GDK, most people never break out to X11 unless they absolute have to. The Cairo API is nice enough though that GDK doesn't even attempt to introduce wrappers. If you muck with X11, accessing the same info with GDK might prove harmful.
One problem is that this is libvirt version specific. For example, libvirt x doesn't support spice so we control that thorough qmp. But libvirt x+1 does support spice and now it gets confused about all the spice messages.
That's only a problem if we only support a single QMP session. This is exactly why we need to support multiple QMP sessions (and do). Regards, Anthony Liguori
On 03/23/2010 09:31 PM, Anthony Liguori wrote:
One problem is that this is libvirt version specific. For example, libvirt x doesn't support spice so we control that thorough qmp. But libvirt x+1 does support spice and now it gets confused about all the spice messages.
That's only a problem if we only support a single QMP session. This is exactly why we need to support multiple QMP sessions (and do).
It's unrelated to the number of sessions. libvirt expects state that it manages in qemu not to change randomly. Users know that, so they will only manage non-libvirt state in their private session. But a new version of libvirt may expand its scope and start managing this area, leading to conflicts. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Avi Kivity wrote:
On 03/23/2010 09:31 PM, Anthony Liguori wrote:
One problem is that this is libvirt version specific. For example, libvirt x doesn't support spice so we control that thorough qmp. But libvirt x+1 does support spice and now it gets confused about all the spice messages.
That's only a problem if we only support a single QMP session. This is exactly why we need to support multiple QMP sessions (and do).
It's unrelated to the number of sessions. libvirt expects state that it manages in qemu not to change randomly. Users know that, so they will only manage non-libvirt state in their private session. But a new version of libvirt may expand its scope and start managing this area, leading to conflicts.
We have events when state changes, for libvirt to be informed. But that's prone to race conditions. A fairly universal solution to that is "pre"-events, where something that's going to change state emits a pre-event, waits for an ack from all listeners, then proceeds or not according to the acks, then finally emits the post-event to say the state is changed. Between pre and post events, as far as the listener is concerned, the state is uncertain and it must query or wait when it needs the value.(*) In other words, libvirt would register to listen to all the pre-events for state that it cares about being synchronised with. When libvirt expands its scope, it would simply listen to some more. -- Jamie (*) Yes, this is like 2-phase transactions, and it's also a bit like MESI caching. But it's easier to understand than both :-)
participants (12)
-
Alexander Graf -
Andi Kleen -
Anthony Liguori -
Avi Kivity -
Cole Robinson -
Daniel P. Berrange -
Daniel Veillard -
Gerd Hoffmann -
Jamie Lokier -
Jes Sorensen -
Juan Quintela -
Markus Armbruster