[libvirt] libguestfs integration: rich disk access for libvirt applications
Libguestfs provides very nice functionality for applications that need to work with disk images. The includes provisioning applications that set up or customize disk images. It also includes backup applications that want to look inside disk image snapshots - both at the block and file level. What's missing for libguestfs to fill this role is integration that allows libvirt and libguestfs to work together with the same network-transparent remote API approach. In the past we have discussed remoting libguestfs and Richard presented possible approaches: https://www.redhat.com/archives/libguestfs/2011-May/msg00093.html Could libvirt could provide a secure channel over which the libguestfs client that an application is linked against talks to the remote guestfsd? Would libguestfs need some new parameters to connect to a remote libvirtd and create its appliance VM? In terms of application use cases, I'm thinking along the lines of using libvirt to enumerate storage volumes and then switching to libguestfs to actually access the storage volume remotely. Is there enough information exposed by libvirt today to switch over to libguestfs and point it at the storage volume/image file? Stefan
On Tue, Sep 27, 2011 at 10:10:00AM +0100, Stefan Hajnoczi wrote:
Libguestfs provides very nice functionality for applications that need to work with disk images. The includes provisioning applications that set up or customize disk images. It also includes backup applications that want to look inside disk image snapshots - both at the block and file level.
What's missing for libguestfs to fill this role is integration that allows libvirt and libguestfs to work together with the same network-transparent remote API approach.
In the past we have discussed remoting libguestfs and Richard presented possible approaches: https://www.redhat.com/archives/libguestfs/2011-May/msg00093.html
Could libvirt could provide a secure channel over which the libguestfs client that an application is linked against talks to the remote guestfsd?
Would libguestfs need some new parameters to connect to a remote libvirtd and create its appliance VM?
In terms of application use cases, I'm thinking along the lines of using libvirt to enumerate storage volumes and then switching to libguestfs to actually access the storage volume remotely. Is there enough information exposed by libvirt today to switch over to libguestfs and point it at the storage volume/image file?
I know that Dan had some ideas in this area, so thanks for CC-ing him. Ideally what we'd like to have happen is that the existing tools just work for remote domains, eg: virt-edit -d Guest /etc/motd virt-edit -c qemu+ssh://remote/system -d Guest /etc/motd would both just work without the user needing to do anything special. How to make that happen is the complicated bit ... Currently what the '-d' option ('guestfs_add_domain' API) does is it grabs the guest libvirt XML, parses out the <disk> element, and looks for the path to the disk. We then just open this path as a local file. In the remote case, one of three things can happen: - path doesn't exist so we fail (the usual case) - path does exist because the user has arranged for paths to be available on the local machine (think: SAN) - we open some other unrelated disk image (oops) Libvirt isn't giving us any indication that the guest is located remotely. What would be nice would be that libvirtd [which is on the same host as the guest always] opens the disk image for us and by some method forwards this back to us. Ideally only in the remote case, so that we can work as efficiently as possible in the local case. I can see several ways this could work: (1) We use the existing virDomainBlockPeek API. libvirt already has an API for peeking into a remote disk. It's inefficient and is limited to max 64K read per message, and writing is not possible. qemu would need to be modified to be able to access this API (or we could write some sort of libvirt<->nbd translation layer). http://libvirt.org/html/libvirt-libvirt.html#virDomainBlockPeek (2) libvirtd runs nbd-server and forwards this back to us. How would qemu access this on the client side? (3) libvirtd runs the libguestfs appliance. It forwards the socket back to us over the libvirt connection. I think this is what Dan had in mind, and it's reasonably easy to integrate this into both libvirtd and libguestfs. (4) libvirt somehow gains the whole libguestfs API and forwards API calls. Others ways ...? What's the next step to make this happen? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Tue, Sep 27, 2011 at 10:34:19AM +0100, Richard W.M. Jones wrote:
(3) libvirtd runs the libguestfs appliance. It forwards the socket back to us over the libvirt connection. I think this is what Dan had in mind, and it's reasonably easy to integrate this into both libvirtd and libguestfs.
[To explain this option in more detail] (a) Modify guestfs_add_domain in the case where the libvirt connection is remote. How do we know if the libvirt connection is remote? https://www.redhat.com/archives/libvir-list/2011-April/msg00890.html As Dan pointed out elsewhere, any libvirt URI that has a host element is probably "remote" even if the host element is "localhost" (since that might indicate some sort of cross-user/-policy access). (b) Add a libvirt API to open up a remote libguestfs. Something like this: int virConnectStartGuestfsDaemon ( virConnectPtr conn, /* libvirt connection */ const char *unix_path, /* path to local unix domain socket */ unsigned int flags ); (c) At the remote end, libvirtd launches the libguestfs appliance in qemu. - How would it do this? - Would it reuse libguestfs code for this? - Would it call febootstrap-supermin-helper itself? Or another method? - Do we need to pass any libguestfs parameters over (eg. path? qemu? verbose? append? memsize? selinux?) - Where do qemu log messages go to? (d) The virtio serial port is forwarded back to the libvirt client side, causing the Unix domain socket to be opened. (e) The libguestfs library now calls guestfs_set_attach_method with the name of the socket. (f) The libguestfs tool works away as before. (No change to the tool or API, which is good). (g) guestfs_close closes the socket. - Does libvirt detect this automatically? - Do we need to take some action to knock down the forwarding? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Tue, Sep 27, 2011 at 12:21:05PM +0100, Richard W.M. Jones wrote:
On Tue, Sep 27, 2011 at 10:34:19AM +0100, Richard W.M. Jones wrote:
(3) libvirtd runs the libguestfs appliance. It forwards the socket back to us over the libvirt connection. I think this is what Dan had in mind, and it's reasonably easy to integrate this into both libvirtd and libguestfs.
[To explain this option in more detail]
(a) Modify guestfs_add_domain in the case where the libvirt connection is remote. How do we know if the libvirt connection is remote?
https://www.redhat.com/archives/libvir-list/2011-April/msg00890.html
As Dan pointed out elsewhere, any libvirt URI that has a host element is probably "remote" even if the host element is "localhost" (since that might indicate some sort of cross-user/-policy access).
(b) Add a libvirt API to open up a remote libguestfs. Something like this:
int virConnectStartGuestfsDaemon ( virConnectPtr conn, /* libvirt connection */ const char *unix_path, /* path to local unix domain socket */ unsigned int flags );
This is fairly similar to what I just proposed, but I suggested just directly returning a guestfs_h handle, so that we do't have to hardcode use of a UNIX domain socket.
(c) At the remote end, libvirtd launches the libguestfs appliance in qemu.
- How would it do this? - Would it reuse libguestfs code for this?
We would want to ultimately call into virDomainCreate() so that we ensure integration with SELinux/lock manager/disk secrets, etc. So I wouldn't want to use the libguestfs code for spawning QEMU.
- Would it call febootstrap-supermin-helper itself? Or another method?
Yeah, I imagine we would call that before launching the guest to get out kernel/initrd/whatever else it outputs.
- Do we need to pass any libguestfs parameters over (eg. path? qemu? verbose? append? memsize? selinux?)
Ideally all those could be determined by libvirtd, but perhaps a couple of them would be exposed as flags.
- Where do qemu log messages go to?
/var/log/libvirt/qemu/$GUEST.log which reminds me that we need to add an API virDomainOpenLog(virDomainPtr, virStreamPtr) to let an mgmt app remotely read the QEMU log.
(d) The virtio serial port is forwarded back to the libvirt client side, causing the Unix domain socket to be opened.
(e) The libguestfs library now calls guestfs_set_attach_method with the name of the socket.
In my proposal those would all be done by the virConnectStartGuestfsDaemon API....
(f) The libguestfs tool works away as before. (No change to the tool or API, which is good).
...but this would require changes to any existing apps to use the new API.
(g) guestfs_close closes the socket.
- Does libvirt detect this automatically? - Do we need to take some action to knock down the forwarding?
With transient guests, you can now set a flag to put them into what we call an 'auto destroy' mode. This means when the last reference to the virConnectPtr is released, libvirt will kill the guest. If guestfs wanted to start multiple appliances with the same virConnecPtr then this would not be sufficient and some other form of cleanup would be needed. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, Sep 27, 2011 at 12:21:05PM +0100, Richard W.M. Jones wrote:
On Tue, Sep 27, 2011 at 10:34:19AM +0100, Richard W.M. Jones wrote:
(3) libvirtd runs the libguestfs appliance. It forwards the socket back to us over the libvirt connection. I think this is what Dan had in mind, and it's reasonably easy to integrate this into both libvirtd and libguestfs.
[To explain this option in more detail]
(a) Modify guestfs_add_domain in the case where the libvirt connection is remote. How do we know if the libvirt connection is remote?
https://www.redhat.com/archives/libvir-list/2011-April/msg00890.html
As Dan pointed out elsewhere, any libvirt URI that has a host element is probably "remote" even if the host element is "localhost" (since that might indicate some sort of cross-user/-policy access).
(b) Add a libvirt API to open up a remote libguestfs. Something like this:
int virConnectStartGuestfsDaemon ( virConnectPtr conn, /* libvirt connection */ const char *unix_path, /* path to local unix domain socket */ unsigned int flags );
(c) At the remote end, libvirtd launches the libguestfs appliance in qemu.
- How would it do this? - Would it reuse libguestfs code for this? - Would it call febootstrap-supermin-helper itself? Or another method? - Do we need to pass any libguestfs parameters over (eg. path? qemu? verbose? append? memsize? selinux?) - Where do qemu log messages go to?
(d) The virtio serial port is forwarded back to the libvirt client side, causing the Unix domain socket to be opened.
(e) The libguestfs library now calls guestfs_set_attach_method with the name of the socket.
Since you don't mind using libvirt APIs internally if they #ifdef'able, then how practical is it to change the I/O routines inside libguestfs to optionally use virStreamRecv/virStreamSend directly, instead of having to proxy the stream via a client side UNIX socket ? Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, Sep 27, 2011 at 10:10:00AM +0100, Stefan Hajnoczi wrote:
Libguestfs provides very nice functionality for applications that need to work with disk images. The includes provisioning applications that set up or customize disk images. It also includes backup applications that want to look inside disk image snapshots - both at the block and file level.
What's missing for libguestfs to fill this role is integration that allows libvirt and libguestfs to work together with the same network-transparent remote API approach.
In the past we have discussed remoting libguestfs and Richard presented possible approaches: https://www.redhat.com/archives/libguestfs/2011-May/msg00093.html
Could libvirt could provide a secure channel over which the libguestfs client that an application is linked against talks to the remote guestfsd?
Would libguestfs need some new parameters to connect to a remote libvirtd and create its appliance VM?
In terms of application use cases, I'm thinking along the lines of using libvirt to enumerate storage volumes and then switching to libguestfs to actually access the storage volume remotely. Is there enough information exposed by libvirt today to switch over to libguestfs and point it at the storage volume/image file?
IMHO, the overall goal for integration is that anytime you have a libvirt connection, you should be able to use libguestfs to access storage volumes or guests without further remote configuration required. The primary way to achieve this is if all communication takes place over the existing libvirt data transport, and not any out of band transport (NBD, NFS, SSH, whatever) The next obvious question is which side of the libvirt connection should the guestfs daemon/appliance be running. There are 3 main scenarios wrt guests 1. Guest is running and has a guestfs channel already present 2. Guest is running and does not have a guestfs channel present 3. Guest is not running. In case 1, obviously the daemon is running server side in the main guest OS. In case 2, you could either boot the guestfs appliance readonly, or hotplug a virtio channel for guestfs into the running guest to get readwrite access (assumes the guest OS has neccessary magic to auto-launch the daemon). The latter is obviously server side again, the former could be client or server side. In case 3, you need to boot the appliance. This could be client or server side. To run the appliance server side, will require that libvirtd gains the ability to spawn a guest with the appliance, and tunnel the host side of the virtio channel back over the libvirt connection to the client. On the client side, either libguestfs needs to have a set of pluggable I/O functions, which we can redirect to use the virStreamPtr APIs, or libvirt would have to turn its stream back into a UNIX socket. The latter is doable without any more libguestfs changes, but introduces extra I/O copying, while the former is most effecient, but requires libguestfs changes. To run the appliance client side, will require that libvirtd gains the ability to tunnel disk access over the libvirt connection to the client, which then runs the appliance somehow. On the client side, either QEMU needs a block driver with a set of pluggable I/O functions, which we can redirect to use libvirt APIs, or libvirt would have to turn its stream back into a UNIX socket running NBD protocol, or a create a userspace block device (FUSE but for block devs). Again the latter is doable without any QEMU changes, but has extra I/O copying, whjile the former is most efficient. Finally, there are some other things worth considering where libguestfs currently has functionality gaps wrt libvirt. - Integration with the libvirt lock manager infrastructure to prevent concurrent R/W guest disk access - Integration with the libvirt secret manager infrastructure to allow access to encrypted QCow2 disks - Integration with sVirt to ensure the appliance runs in a strictly confined context - Hotplug to allow extra disks to be inserted/removed to/from a running libguestfs appliance Running the appliance server side, spawned by libvirt would allow all of those points to be satisfied. There is a final problem in that not all hypervisors feature libvirtd, eg VMWare ESX, Hyper V and VirtualBox. Some of them might expose guest disks via HTTP or some other protocols that QEMU might be able to access directly. Others though would require that you setup some kind of shared filesystem (eg mount NFS) to access them. Others might let you SSH in (with suitable credentials) which lets you use FUSE SSHFS. With such hypervisors it is more or less impossible to satisfy my initial requirement that 'any time you have a libvirt connection you can run libguestfs without further admin configuration'. Probably the best you can do here is to ensure that there is one API you use for access guest disks. One other point worth mentioning is that libguestfs.so does not want to directly link to libvirt.so, and vica-verca, to ensure we both avoid pulling major new dependancy chains for all users. Similarly, if at all possible, any existing libguestfs application would like to be able to 'just work' with any libvirt integration without further code changes. Now what do I think we should do. Personally I would really like to have libguestfs be integrating with the lock manager, secret manager and sVirt infrastructure in libvirt. The only real practical way for this to happen is if the appliance runs server side, spawned via the normal QEMU driver guest startup code in libvirt. So I discount any idea of running the appliance client side & tunnelling block devices over libvirt. Also note that different solutions may be required for hypervisors without libvirt. I am ignoring such hypervisors for now. If I were ignoring the requirement that libguestfs does not link to libvirt, then you could quite likely make all this happen with only a simple additional API in libvirt. We need an API to let a client open a connection to a <channel> device, using the virStreamPtr API. If the guests were not running, libguestfs would use virDomainCreate to spawn a transient, auto-detroy guest, with a custom kernel/initrd that runs the appliance, and an additional <channel> device, but with all other parts of the guest XML unchanged. This would ensure all the lock manager, sVirt and secret stuff 'just works'. If the guest is already running, libguestfs would just query the XML to find the <channel> device configuration. Then it could just use a new API like virDomainOpenChannel(virStreamPtr, const char *channelid) to get a stream to talk to the guestfs daemon with. If libguestfs were just wanting to use a random disk device, not yet associated with any guest, then it again would use virDOmainCreate to spawn a transient, auto-destroy guest, with an XML config it fully creates. This would all make it possible for any guestfs based application to work with libvirt without any app code changes. I assume it is still the case, however, that libguestfs does *not* want to link directly to libvirt and use it for guest creation. Thus my idea is to find an alternative solution that gets as close to that "ideal" setup as possible. To do this I would create what I call a bridging library, to be named 'libvirt-guestfs.so'. This would have a handful of API calls for handling the initial creation/startup of the appliance & access to the vmchannel device only, delegating everything else to normal libguestfs APIs. - int virDomainStartGuestFS(virDomainPtr dom, int flags) If the guest 'dom' is not already running, boot the guest pointing it at the libguestfs kernel/initrd + vmchannel device. - guestfs_h *virDomainOpenGuestFS(virDomainPtr dom) Server side, find the vmchannel device for the guest, open a stream for it. On the client side the stream would somehow be proxied to a UNIX domain socket. We then call the libguestfs APIs to neccessary to attach the external UNIX domain socket, create the guestfs_h handle and return it. With those two APIs (possibly 1 or 2 more), an application wanting to use an integrated libguestfs+libvirt, would use libvirt-guestfs.so to obtain their guestfs_h * handle, and then use all the other libguestfs.so APIs as normal. This avoids having to wrap every single libguestfs API in libvirt. For apps like virt-manager this would easily be workable, other existing libguestfs apps would however need to have some small changes in their initial connection setup code to optionally use libvirt-guestfs.so Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, Sep 27, 2011 at 12:20:31PM +0100, Daniel P. Berrange wrote:
One other point worth mentioning is that libguestfs.so does not want to directly link to libvirt.so, and vica-verca, to ensure we both avoid pulling major new dependancy chains for all users.
Actually libguestfs.so in Fedora links to libvirt.so today. What we don't want is libvirt to be required *for libguestfs to compile*. At the moment if libvirt is not available, we disable one API at compile time (using #ifdef HAVE_LIBVIRT etc). I don't this discussion is affected by this.
If I were ignoring the requirement that libguestfs does not link to libvirt, then you could quite likely make all this happen with only a simple additional API in libvirt. We need an API to let a client open a connection to a <channel> device, using the virStreamPtr API.
If the guests were not running, libguestfs would use virDomainCreate to spawn a transient, auto-detroy guest, with a custom kernel/initrd that runs the appliance, and an additional <channel> device, but with all other parts of the guest XML unchanged. This would ensure all the lock manager, sVirt and secret stuff 'just works'. If the guest is already running, libguestfs would just query the XML to find the <channel> device configuration. Then it could just use a new API like virDomainOpenChannel(virStreamPtr, const char *channelid) to get a stream to talk to the guestfs daemon with.
I'm with you up to here, but there's a practical problem: How do we create the appliance kernel/initrd/root disk on the server side? (I'm assuming that libvirt doesn't forward these large objects from the client to the server.) Normally these objects are created by running febootstrap-supermin-helper.
To do this I would create what I call a bridging library, to be named 'libvirt-guestfs.so'.
See above, although we have converged on similar designs, but for different reasons. FWIW as outlined in the other email, I think we can do this without a bridging library, and just making changes behind the scenes in guestfs_add_domain[1], which would be transparent to callers. Rich. [1] http://git.annexia.org/?p=libguestfs.git;a=blob;f=src/virt.c;hb=HEAD -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
On Tue, Sep 27, 2011 at 12:35:21PM +0100, Richard W.M. Jones wrote:
On Tue, Sep 27, 2011 at 12:20:31PM +0100, Daniel P. Berrange wrote:
One other point worth mentioning is that libguestfs.so does not want to directly link to libvirt.so, and vica-verca, to ensure we both avoid pulling major new dependancy chains for all users.
Actually libguestfs.so in Fedora links to libvirt.so today. What we don't want is libvirt to be required *for libguestfs to compile*.
At the moment if libvirt is not available, we disable one API at compile time (using #ifdef HAVE_LIBVIRT etc). I don't this discussion is affected by this.
If I were ignoring the requirement that libguestfs does not link to libvirt, then you could quite likely make all this happen with only a simple additional API in libvirt. We need an API to let a client open a connection to a <channel> device, using the virStreamPtr API.
If the guests were not running, libguestfs would use virDomainCreate to spawn a transient, auto-detroy guest, with a custom kernel/initrd that runs the appliance, and an additional <channel> device, but with all other parts of the guest XML unchanged. This would ensure all the lock manager, sVirt and secret stuff 'just works'. If the guest is already running, libguestfs would just query the XML to find the <channel> device configuration. Then it could just use a new API like virDomainOpenChannel(virStreamPtr, const char *channelid) to get a stream to talk to the guestfs daemon with.
I'm with you up to here, but there's a practical problem: How do we create the appliance kernel/initrd/root disk on the server side? (I'm assuming that libvirt doesn't forward these large objects from the client to the server.) Normally these objects are created by running febootstrap-supermin-helper.
Yeah, I think libvirt will just have to be able to run the febootstrap-supermin-helper program itself at the appropriate time. As long as we create some SELinux security policy to confine the febootstrap-supermin-helper I don't see that as a very significant problem. I think the SELinux policy would be quite straightforward to create, since the program has a pretty well defined single task to perform
To do this I would create what I call a bridging library, to be named 'libvirt-guestfs.so'.
See above, although we have converged on similar designs, but for different reasons.
FWIW as outlined in the other email, I think we can do this without a bridging library, and just making changes behind the scenes in guestfs_add_domain[1], which would be transparent to callers.
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
To put this all into one place: (1) An ugly new libvirt API that runs febootstrap-supermin-helper to create the appliance. (2) Modify guestfs_add_domain to launch a transient domain using virDomainCreate. (Forget about the live case for the moment). (3) A new libvirt API to open and forward a virtio-serial channel. This seems generally a useful thing. int virDomainOpenChannel(virStreamPtr, const char *channelid); (4) libguestfs somehow has to access a virStreamPtr. We can work this one out. (5) libvirt will autoclose the domain for us when we close the virConnectPtr. I'm worried about item (1) in this list ... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
On Tue, Sep 27, 2011 at 12:55:19PM +0100, Richard W.M. Jones wrote:
To put this all into one place:
(1) An ugly new libvirt API that runs febootstrap-supermin-helper to create the appliance.
(2) Modify guestfs_add_domain to launch a transient domain using virDomainCreate. (Forget about the live case for the moment).
(3) A new libvirt API to open and forward a virtio-serial channel. This seems generally a useful thing.
int virDomainOpenChannel(virStreamPtr, const char *channelid);
(4) libguestfs somehow has to access a virStreamPtr. We can work this one out.
(5) libvirt will autoclose the domain for us when we close the virConnectPtr.
This all sounds like a good plan to aim for. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, Sep 27, 2011 at 12:55 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
To put this all into one place:
(1) An ugly new libvirt API that runs febootstrap-supermin-helper to create the appliance. [...] I'm worried about item (1) in this list ...
This is the only instance where libvirt knows about libguestfs. All other steps are libguest only or involve libguestfs knowing about libvirt. Would it be possible introduce a "domain-builder" concept into libvirt? When libguestfs is installed it drops a domain-builder configuration/script that libvirt can pick up. Then you can say something like virDomainBuild(name="guestfs-appliance", builder="guestfs"). Does febootstrap-supermin-helper need to be dynamic or could libguestfs create a /var/lib/guestfs/appliance-initramfs.gz on install? Then libguestfs on the client can create the appliance domain and point at that static initramfs file path. Stefan
On Wed, Sep 28, 2011 at 11:14:57AM +0100, Stefan Hajnoczi wrote:
Does febootstrap-supermin-helper need to be dynamic or could libguestfs create a /var/lib/guestfs/appliance-initramfs.gz on install? Then libguestfs on the client can create the appliance domain and point at that static initramfs file path.
This is how the Debian package of libguestfs works (Hilko's official package, not my one). However this is troublesome because it means any security problem in a dependent program is baked into the appliance. Applying a security update to the host wouldn't update this libguestfs appliance. Compare this to the way febootstrap-supermin-helper normally works (eg upstream, Fedora and RHEL): the appliance is rebuilt whenever any change is noticed in a dependent program. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On Wed, Sep 28, 2011 at 1:19 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Sep 28, 2011 at 11:14:57AM +0100, Stefan Hajnoczi wrote:
Does febootstrap-supermin-helper need to be dynamic or could libguestfs create a /var/lib/guestfs/appliance-initramfs.gz on install? Then libguestfs on the client can create the appliance domain and point at that static initramfs file path.
This is how the Debian package of libguestfs works (Hilko's official package, not my one).
However this is troublesome because it means any security problem in a dependent program is baked into the appliance. Applying a security update to the host wouldn't update this libguestfs appliance. Compare this to the way febootstrap-supermin-helper normally works (eg upstream, Fedora and RHEL): the appliance is rebuilt whenever any change is noticed in a dependent program.
That sounds like a limitation in the packaging system. If 'watch' hooks can be registered by the libguestfs package on its dependencies, then it can rebuild itself every thing a dependency changes. Or the low-tech way is for the libguestfs package maintainer to create a new package each time its dependencies have updated - Debian has a volatile repo for packages that change a lot. At the end of the day we have this problem because the libguestfs appliance is a distro built from the underlying distro itself :)! Stefan
On Wed, Sep 28, 2011 at 05:35:46PM +0100, Stefan Hajnoczi wrote:
On Wed, Sep 28, 2011 at 1:19 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Sep 28, 2011 at 11:14:57AM +0100, Stefan Hajnoczi wrote:
Does febootstrap-supermin-helper need to be dynamic or could libguestfs create a /var/lib/guestfs/appliance-initramfs.gz on install? Then libguestfs on the client can create the appliance domain and point at that static initramfs file path.
This is how the Debian package of libguestfs works (Hilko's official package, not my one).
However this is troublesome because it means any security problem in a dependent program is baked into the appliance. Applying a security update to the host wouldn't update this libguestfs appliance. Compare this to the way febootstrap-supermin-helper normally works (eg upstream, Fedora and RHEL): the appliance is rebuilt whenever any change is noticed in a dependent program.
That sounds like a limitation in the packaging system.
If 'watch' hooks can be registered by the libguestfs package on its dependencies, then it can rebuild itself every thing a dependency changes. Or the low-tech way is for the libguestfs package maintainer to create a new package each time its dependencies have updated - Debian has a volatile repo for packages that change a lot.
At the end of the day we have this problem because the libguestfs appliance is a distro built from the underlying distro itself :)!
RPM & dpkg both have trigger mechanisms. The Debian package doesn't appear to use it for whatever reason. In the RPM we just don't use triggers because the checksum method we're using is more convenient and produces about the same result. We could change this but I want to look at other alternatives as well. In particular, using 9pfs might mean there's no need to explicitly build a root appliance at all (but it needs some qemu changes). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
On Wed, Sep 28, 2011 at 11:14:57AM +0100, Stefan Hajnoczi wrote:
On Tue, Sep 27, 2011 at 12:55 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
To put this all into one place:
(1) An ugly new libvirt API that runs febootstrap-supermin-helper to create the appliance. [...] I'm worried about item (1) in this list ...
This is the only instance where libvirt knows about libguestfs. All other steps are libguest only or involve libguestfs knowing about libvirt.
Would it be possible introduce a "domain-builder" concept into libvirt? When libguestfs is installed it drops a domain-builder configuration/script that libvirt can pick up. Then you can say something like virDomainBuild(name="guestfs-appliance", builder="guestfs").
We do have a historical syntax from Xen paravirt which lets us call out to a helper at boot time, namely the "<bootloader>" element. With Xen this is typically something like pygrub, or pxegrub, which does some work and writes out a kernel+initrd into temporary files, and prints the file paths + any kernel args on stdout. We could just wire up this concept in KVM too without any real trouble, and then we could have guestfs-bootloader script todo the magic setup Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, Sep 28, 2011 at 06:37:17PM +0100, Daniel P. Berrange wrote:
On Wed, Sep 28, 2011 at 11:14:57AM +0100, Stefan Hajnoczi wrote:
On Tue, Sep 27, 2011 at 12:55 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
To put this all into one place:
(1) An ugly new libvirt API that runs febootstrap-supermin-helper to create the appliance. [...] I'm worried about item (1) in this list ...
This is the only instance where libvirt knows about libguestfs. All other steps are libguest only or involve libguestfs knowing about libvirt.
Would it be possible introduce a "domain-builder" concept into libvirt? When libguestfs is installed it drops a domain-builder configuration/script that libvirt can pick up. Then you can say something like virDomainBuild(name="guestfs-appliance", builder="guestfs").
We do have a historical syntax from Xen paravirt which lets us call out to a helper at boot time, namely the "<bootloader>" element. With Xen this is typically something like pygrub, or pxegrub, which does some work and writes out a kernel+initrd into temporary files, and prints the file paths + any kernel args on stdout.
We could just wire up this concept in KVM too without any real trouble, and then we could have guestfs-bootloader script todo the magic setup
I'm fine with this. Are there security implications to allowing users to add <bootloader> clauses pointing at random scripts that get run on remote machines as different users? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On 09/28/2011 11:52 AM, Richard W.M. Jones wrote:
We do have a historical syntax from Xen paravirt which lets us call out to a helper at boot time, namely the "<bootloader>" element. With Xen this is typically something like pygrub, or pxegrub, which does some work and writes out a kernel+initrd into temporary files, and prints the file paths + any kernel args on stdout.
We could just wire up this concept in KVM too without any real trouble, and then we could have guestfs-bootloader script todo the magic setup
I'm fine with this.
Are there security implications to allowing users to add<bootloader> clauses pointing at random scripts that get run on remote machines as different users?
No more so than the fact that we let random clients specify <disk> devices to random devices on remote machines. Right now, granting non-read-only connection rights to a user effectively gives them root access to the machine. There's eventual plans to further restrict things via per-command ACLs, and this should be considered during those plans, but until then, I don't see it as any larger a hole than anything else already present in libvirt design. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org
On Wed, Sep 28, 2011 at 06:52:13PM +0100, Richard W.M. Jones wrote:
On Wed, Sep 28, 2011 at 06:37:17PM +0100, Daniel P. Berrange wrote:
On Wed, Sep 28, 2011 at 11:14:57AM +0100, Stefan Hajnoczi wrote:
On Tue, Sep 27, 2011 at 12:55 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
To put this all into one place:
(1) An ugly new libvirt API that runs febootstrap-supermin-helper to create the appliance. [...] I'm worried about item (1) in this list ...
This is the only instance where libvirt knows about libguestfs. All other steps are libguest only or involve libguestfs knowing about libvirt.
Would it be possible introduce a "domain-builder" concept into libvirt? When libguestfs is installed it drops a domain-builder configuration/script that libvirt can pick up. Then you can say something like virDomainBuild(name="guestfs-appliance", builder="guestfs").
We do have a historical syntax from Xen paravirt which lets us call out to a helper at boot time, namely the "<bootloader>" element. With Xen this is typically something like pygrub, or pxegrub, which does some work and writes out a kernel+initrd into temporary files, and prints the file paths + any kernel args on stdout.
We could just wire up this concept in KVM too without any real trouble, and then we could have guestfs-bootloader script todo the magic setup
I'm fine with this.
Are there security implications to allowing users to add <bootloader> clauses pointing at random scripts that get run on remote machines as different users?
Yes, but you have to consider a connection to libvirtd, to be equivalent to a root shell at this time anyway. When we get RBAC in libvirt we'll be able to control who can make such configurations, and/or whitelist bootloaders in the SELinux policy so only trusted ones can be run Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, Sep 28, 2011 at 06:37:17PM +0100, Daniel P. Berrange wrote:
We do have a historical syntax from Xen paravirt which lets us call out to a helper at boot time, namely the "<bootloader>" element. With Xen this is typically something like pygrub, or pxegrub, which does some work and writes out a kernel+initrd into temporary files, and prints the file paths + any kernel args on stdout.
We could just wire up this concept in KVM too without any real trouble, and then we could have guestfs-bootloader script todo the magic setup
The attached scripts show how it would work, and furthermore they show that it really does work. The first script is the bootloader script. This creates the libguestfs appliance (3 files called /tmp/kernel, /tmp/initrd and /tmp/root). At the moment, no caching of the appliance is implemented, but that is easy to add. The second script shows the sort of changes we would make to the guestfs_add_domain API in order to run the appliance from libvirt. It's a Ruby script that creates a transient domain, booting from the appliance created in step 1, with the correct virtio-serial port etc. It then tells libguestfs to connect to the virtio-serial socket. Finally it issues some libguestfs commands to create a filesystem, so you can tell that it is actually talking to the libguestfs daemon. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
participants (4)
-
Daniel P. Berrange -
Eric Blake -
Richard W.M. Jones -
Stefan Hajnoczi