5:48 p.m.
On Wed, Jan 24, 2007 at 02:17:31PM +0000, Richard W.M. Jones wrote:
This is a follow on to this thread:
https://www.redhat.com/archives/libvir-list/2007-January/thread.html#00064
but I think it deserves a thread of its own for discussion.
Background:
Dan drew this diagram proposing a way to include remote access to
systems from within libvirt:
http://people.redhat.com/berrange/libvirt/libvirt-arch-remote-2.png
libvirt would continue as now to provide direct hypervisor calls,
direct access to xend and so on. But in addition, a new backend
would be written ("remote") which could talk to a remote daemon
("libvirtd") using some sort of RPC mechanism.
Position:
I gave this architecture some thought over the weekend, and I
like it for the following reasons (some not very technical):
* Authentication and encryption is handled entirely within the
libvirt / libvirtd library, allowing us to use whatever RPC
mechanism we like on top of a selection of transports of our
choosing (eg. GnuTLS, ssh, unencrypted TCP sockets, ...)
Yes, having a single consistent wire encyption & user auth system across
all virt backends makes a very nice end user / admin story
* We don't need to modify xend at all, and additionally we
won't
need to modify future flavour-of-the-month virtual machine monitors.
I have a particular issue with xend (written in Python) because
in my own tests I've seen my Python XMLRPC/SSL server
actually segfault. It doesn't inspire me that this Python
solution is adding anything more than apparent security.
Did I mention XenD is slow. If we can get remote management of Xen
bypassing XenD just like we do for the local case, we'll be much
better off.
* The architecture is very flexible: It allows virt-manager to
run as root or as non-root, according to customer wishes.
virt-manager can make direct HV calls, or everything can be
remoted, and it's easy to explain to the user about the
performance vs management trade-offs.
* It's relatively easy to implement. Note that libvirtd is just
a thin server layer linked to its own copy of libvirt.
* Another proposal was to make all libvirt calls remote
(http://people.redhat.com/berrange/libvirt/libvirt-arch-remote-3.png)
but I don't think this is a going concern because (1) it requires
a daemon always be run, which is another installation problem and
another chance for sysadmins to give up, and (2) the perception will
be that this is slow, whether or not that is actually true.
I'd never compared performance of direct hypercalls vs libvirt_proxy
before, so I did a little test. The most commonly called method is
virt-manager is virDomainGetInfo for fetching current status of a
running domain - we call that once a second per guest.
So I wrote a simple program in C which calls virDomainGetInfo 100,000
times for 3 active guest VMs. I ran the test under a couple of different
libvirt backends. The results were:
1. As root, direct hypercalls -> 1.4 seconds
2. As non-root, hypercalls via libvirt_proxy -> 9 seconds
3. As non-root, via XenD -> 45 minutes [1]
So although it is x10 slower than hypercalls, the libvirt_proxy is
actaully pretty damn fast - 9 seconds for 300,000 calls.
There are many reasons the XenD path is slow. Each operation makes
a new HTTP request. It spawns a new thread per request. It talks to
XenStore for every request which has very high I/O overhead. It uses
the old SEXPR protocol which requests far more info than we actually
need. It is written in Python. Now I'm sure we can improve performance
somewhat by switching to the new XML-RPC api, and getting persistent
connections running, but I doubt it'll ever be as fast as libvirt_proxy
let alone hypercalls. So as mentioned above, I'd like to take XenD
out of the loop for remote management just like we do for the local
case with libvirt_proxy, but with full authenticated read+write access.
Now some concerns:
* libvirtd will likely need to be run as root, so another root
daemon written in C listening on a public port. (On the other
hand, xend listening on a public port also isn't too desirable,
even with authentication).
For Xen we have no choice but to have something running as root
since hypercalls needs to mlock() memory and access the Xen device
node. So both options are unpleasent, but we have to choose one,
and I can't say XenD is the obvious winner - particularly given the
tendancy of python SSL code to segfault. We can, however, make sure
that libvirtd is written to allow the full suite of modern protection
mechanisms to be applied - SELinux, execshield, TLS, fortify source
etc.
* If Xen upstream in the meantime come up with a secure remote
access
method then potentially this means clients could have to choose
between the two, or run two services (libvirtd + Xen/remote).
* There are issues with versioning the remote API. Do we allow
different versions of libvirt/libvirtd to talk to each other?
Do we provide backwards compatibility when we move to a new API?
We can simply apply the same rules as we do for public API. No changes
to existing calls, only additions are allowed. A simple protocol
version number can allow the client & server to negotiate the mutually
supported feature set.
* Do we allow more than one client to talk to a libvirtd daemon
(no | multiple readers one writer | multiple readers & writers).
The latter - since we have things modifiying domains via XenD, or the
HV indirectly updating domain state, we defacto have multiple writers
already. From my work in the qemu daemon, I didn't encounter any major
problems with allowing multiple writers - by using a poll() based
single-thread event loop approach, I avoided any nasty multi-thread
problems associated with multiple connections. Provided each request
can be completed in a short amount of time, there should be no need to
go fully threaded.
* What's the right level to make a remote API? Should we batch
calls up together?
We may be already constrained by the client side API - all calls in
the libvirt public API are synchronous so from a single client thread
there is nothing available to batch.
I've been investigating RPC mechanisms and there seem to be two
reasonable possibilities, SunRPC and XMLRPC. (Both would need to
run over some sort of secure connection, so there is a layer below
both). My analysis of those is here:
http://et.redhat.com/~rjones/secure_rpc/
SunRPC would handle our current APIs fine. We've talked every now & then
about providing asynchronous callbacks into the API - eg, so the client
can be notified of VM state changes without having to poll the
virDomainGetInfo api every second. The RPC wire protocol certainly
supports that, but its not clear the C apis do.
The XDR wire formating rules are very nicely defined - another option is
to use XDR as the wire encoding for our existing prototype impl in the
qemud.
For XML-RPC I'd like to do a proof of concept of the virDomainGetInfo
method impl to see how much overhead it adds. Hopefully it would be
acceptable, although I'm sure its a fair bit more than XDR / SunRPC.
We would need persistent connections for XML-RPC to be viable, particularly
with TLS enabled. Since XML-RPC doesn't really impose any firm C API
I imagine we could get a-synchronous notifications from the server
working without much trouble.
Dan.
[1] It didn't actually finish after 45 seconds. I just got bored of waiting.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|