3:56 a.m.
On Wed, Jan 24, 2007 at 11:48:47PM +0000, Daniel P. Berrange wrote:
On Wed, Jan 24, 2007 at 02:17:31PM +0000, Richard W.M. Jones wrote:
> * Another proposal was to make all libvirt calls remote
> (http://people.redhat.com/berrange/libvirt/libvirt-arch-remote-3.png)
> but I don't think this is a going concern because (1) it requires
> a daemon always be run, which is another installation problem and
> another chance for sysadmins to give up, and (2) the perception will
> be that this is slow, whether or not that is actually true.
I'd never compared performance of direct hypercalls vs libvirt_proxy
before, so I did a little test. The most commonly called method is
virt-manager is virDomainGetInfo for fetching current status of a
running domain - we call that once a second per guest.
So I wrote a simple program in C which calls virDomainGetInfo 100,000
times for 3 active guest VMs. I ran the test under a couple of different
libvirt backends. The results were:
1. As root, direct hypercalls -> 1.4 seconds
2. As non-root, hypercalls via libvirt_proxy -> 9 seconds
3. As non-root, via XenD -> 45 minutes [1]
So although it is x10 slower than hypercalls, the libvirt_proxy is
actaully pretty damn fast - 9 seconds for 300,000 calls.
Interesting figures, I has expected the proxy inter-process communication
to slow things down more, I guess it works well because scheduling follows
exactly the message passing so there is little latency in the RPC, was that
on a uniprocessor machine ?
There are many reasons the XenD path is slow. Each operation makes
a new HTTP request. It spawns a new thread per request. It talks to
XenStore for every request which has very high I/O overhead. It uses
the old SEXPR protocol which requests far more info than we actually
need. It is written in Python. Now I'm sure we can improve performance
somewhat by switching to the new XML-RPC api, and getting persistent
connections running, but I doubt it'll ever be as fast as libvirt_proxy
let alone hypercalls. So as mentioned above, I'd like to take XenD
out of the loop for remote management just like we do for the local
case with libvirt_proxy, but with full authenticated read+write access.
I love XML, but I doubt switching to XML-RPC will speed things up. Well
maybe if the parser is written in C, but overall parsing an XML instance has
a cost and I doubt you will get anyway close to the 300,000/s of a proxy like
RPC, that would mean 600,000 XML instances parsing per second of just overhead
and that's really not realistic.
My only concern with an ad-hoc protocol like the proxy one is that it
would make harder to build client side say in Java (since we don't have
bindings and that would be a relatively nice way to do it as mixing C
and Java always raises some resistance/problems). Though I really don't
think this is a blocker.
> * If Xen upstream in the meantime come up with a secure remote
access
> method then potentially this means clients could have to choose
> between the two, or run two services (libvirtd + Xen/remote).
>
> * There are issues with versioning the remote API. Do we allow
> different versions of libvirt/libvirtd to talk to each other?
> Do we provide backwards compatibility when we move to a new API?
We can simply apply the same rules as we do for public API. No changes
to existing calls, only additions are allowed. A simple protocol
version number can allow the client & server to negotiate the mutually
supported feature set.
Agreed, actually we already do that with the protocol to the proxy:
....
struct _virProxyPacket {
unsigned short version; /* version of the proxy protocol */
unsigned short command; /* command number a virProxyCommand */
....
If we stick to adding only that should work in general c.f. HTTP 1.0
and HTTP 1.1 long term cohabitation.
> * Do we allow more than one client to talk to a libvirtd
daemon
> (no | multiple readers one writer | multiple readers & writers).
The latter - since we have things modifiying domains via XenD, or the
HV indirectly updating domain state, we defacto have multiple writers
already. From my work in the qemu daemon, I didn't encounter any major
problems with allowing multiple writers - by using a poll() based
single-thread event loop approach, I avoided any nasty multi-thread
problems associated with multiple connections. Provided each request
can be completed in a short amount of time, there should be no need to
go fully threaded.
And if needed we can revamp the design internally later without breaking
any compatibility since we operate in stricly synchronous RPC mode at the
connection level (true from the C API down to the wire).
> * What's the right level to make a remote API? Should we
batch
> calls up together?
We may be already constrained by the client side API - all calls in
the libvirt public API are synchronous so from a single client thread
there is nothing available to batch.
Yup let's avoid the asynch support, that killed so many protocol
that I'm really cautious about this. If that mean we need to add higher
level APIs, why not but this really should come as a user induced evolution,
i.e. field reports :-)
> I've been investigating RPC mechanisms and there seem to be
two
> reasonable possibilities, SunRPC and XMLRPC. (Both would need to
> run over some sort of secure connection, so there is a layer below
> both). My analysis of those is here:
>
> http://et.redhat.com/~rjones/secure_rpc/
SunRPC would handle our current APIs fine. We've talked every now & then
about providing asynchronous callbacks into the API - eg, so the client
can be notified of VM state changes without having to poll the
virDomainGetInfo api every second. The RPC wire protocol certainly
supports that, but its not clear the C apis do.
Callbacks are hairy, somehow I would prefer to allow piggybacking
extra payload on an RPC return than initiating one from the server,
this simplifies things both on the client and server code, but also
integration on the client event loop (please no threads !)
The XDR wire formating rules are very nicely defined - another option
is
to use XDR as the wire encoding for our existing prototype impl in the
qemud.
For XML-RPC I'd like to do a proof of concept of the virDomainGetInfo
method impl to see how much overhead it adds. Hopefully it would be
acceptable, although I'm sure its a fair bit more than XDR / SunRPC.
We would need persistent connections for XML-RPC to be viable, particularly
with TLS enabled. Since XML-RPC doesn't really impose any firm C API
I imagine we could get a-synchronous notifications from the server
working without much trouble.
I'm a bit afraid of the XML-RPC overhead, parsing 10,000 small instances
per second with libxml2 is doable with a bit of tuning but that's still more
than an order of magnitude slower than the ad-hoc protocol and without the
marshalling/demarshalling from the strings values. For SunRPC, well if you
do some testing that will be fantastic.
[1] It didn't actually finish after 45 seconds. I just got bored
of waiting.
s/seconds/minutes/ I guess, and you checked CPU was at 100% not a bad deadlock
somewhere, right ;-) ?
Daniel
--
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard | virtualization library http://libvirt.org/
veillard(a)redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/