On Tue, Sep 27, 2011 at 10:10:00AM +0100, Stefan Hajnoczi wrote: I know that Dan had some ideas in this area, so thanks for CC-ing him. Ideally what we'd like to have happen is that the existing tools just work for remote domains, eg: virt-edit -d Guest /etc/motd virt-edit -c qemu+ssh://remote/system -d Guest /etc/motd would both just work without the user needing to do anything special. How to make that happen is the complicated bit ... Currently what the '-d' option ('guestfs_add_domain' API) does is it grabs the guest libvirt XML, parses out the <disk> element, and looks for the path to the disk. We then just open this path as a local file. In the remote case, one of three things can happen: - path doesn't exist so we fail (the usual case) - path does exist because the user has arranged for paths to be available on the local machine (think: SAN) - we open some other unrelated disk image (oops) Libvirt isn't giving us any indication that the guest is located remotely. What would be nice would be that libvirtd [which is on the same host as the guest always] opens the disk image for us and by some method forwards this back to us. Ideally only in the remote case, so that we can work as efficiently as possible in the local case. I can see several ways this could work: (1) We use the existing virDomainBlockPeek API. libvirt already has an API for peeking into a remote disk. It's inefficient and is limited to max 64K read per message, and writing is not possible. qemu would need to be modified to be able to access this API (or we could write some sort of libvirt<->nbd translation layer). http://libvirt.org/html/libvirt-libvirt.html#virDomainBlockPeek (2) libvirtd runs nbd-server and forwards this back to us. How would qemu access this on the client side? (3) libvirtd runs the libguestfs appliance. It forwards the socket back to us over the libvirt connection. I think this is what Dan had in mind, and it's reasonably easy to integrate this into both libvirtd and libguestfs. (4) libvirt somehow gains the whole libguestfs API and forwards API calls. Others ways ...? What's the next step to make this happen? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

On Tue, Sep 27, 2011 at 12:21:05PM +0100, Richard W.M. Jones wrote: This is fairly similar to what I just proposed, but I suggested just directly returning a guestfs_h handle, so that we do't have to hardcode use of a UNIX domain socket. We would want to ultimately call into virDomainCreate() so that we ensure integration with SELinux/lock manager/disk secrets, etc. So I wouldn't want to use the libguestfs code for spawning QEMU. Yeah, I imagine we would call that before launching the guest to get out kernel/initrd/whatever else it outputs. Ideally all those could be determined by libvirtd, but perhaps a couple of them would be exposed as flags. /var/log/libvirt/qemu/$GUEST.log which reminds me that we need to add an API virDomainOpenLog(virDomainPtr, virStreamPtr) to let an mgmt app remotely read the QEMU log. In my proposal those would all be done by the virConnectStartGuestfsDaemon API.... ...but this would require changes to any existing apps to use the new API. With transient guests, you can now set a flag to put them into what we call an 'auto destroy' mode. This means when the last reference to the virConnectPtr is released, libvirt will kill the guest. If guestfs wanted to start multiple appliances with the same virConnecPtr then this would not be sufficient and some other form of cleanup would be needed. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Tue, Sep 27, 2011 at 10:10:00AM +0100, Stefan Hajnoczi wrote: IMHO, the overall goal for integration is that anytime you have a libvirt connection, you should be able to use libguestfs to access storage volumes or guests without further remote configuration required. The primary way to achieve this is if all communication takes place over the existing libvirt data transport, and not any out of band transport (NBD, NFS, SSH, whatever) The next obvious question is which side of the libvirt connection should the guestfs daemon/appliance be running. There are 3 main scenarios wrt guests 1. Guest is running and has a guestfs channel already present 2. Guest is running and does not have a guestfs channel present 3. Guest is not running. In case 1, obviously the daemon is running server side in the main guest OS. In case 2, you could either boot the guestfs appliance readonly, or hotplug a virtio channel for guestfs into the running guest to get readwrite access (assumes the guest OS has neccessary magic to auto-launch the daemon). The latter is obviously server side again, the former could be client or server side. In case 3, you need to boot the appliance. This could be client or server side. To run the appliance server side, will require that libvirtd gains the ability to spawn a guest with the appliance, and tunnel the host side of the virtio channel back over the libvirt connection to the client. On the client side, either libguestfs needs to have a set of pluggable I/O functions, which we can redirect to use the virStreamPtr APIs, or libvirt would have to turn its stream back into a UNIX socket. The latter is doable without any more libguestfs changes, but introduces extra I/O copying, while the former is most effecient, but requires libguestfs changes. To run the appliance client side, will require that libvirtd gains the ability to tunnel disk access over the libvirt connection to the client, which then runs the appliance somehow. On the client side, either QEMU needs a block driver with a set of pluggable I/O functions, which we can redirect to use libvirt APIs, or libvirt would have to turn its stream back into a UNIX socket running NBD protocol, or a create a userspace block device (FUSE but for block devs). Again the latter is doable without any QEMU changes, but has extra I/O copying, whjile the former is most efficient. Finally, there are some other things worth considering where libguestfs currently has functionality gaps wrt libvirt. - Integration with the libvirt lock manager infrastructure to prevent concurrent R/W guest disk access - Integration with the libvirt secret manager infrastructure to allow access to encrypted QCow2 disks - Integration with sVirt to ensure the appliance runs in a strictly confined context - Hotplug to allow extra disks to be inserted/removed to/from a running libguestfs appliance Running the appliance server side, spawned by libvirt would allow all of those points to be satisfied. There is a final problem in that not all hypervisors feature libvirtd, eg VMWare ESX, Hyper V and VirtualBox. Some of them might expose guest disks via HTTP or some other protocols that QEMU might be able to access directly. Others though would require that you setup some kind of shared filesystem (eg mount NFS) to access them. Others might let you SSH in (with suitable credentials) which lets you use FUSE SSHFS. With such hypervisors it is more or less impossible to satisfy my initial requirement that 'any time you have a libvirt connection you can run libguestfs without further admin configuration'. Probably the best you can do here is to ensure that there is one API you use for access guest disks. One other point worth mentioning is that libguestfs.so does not want to directly link to libvirt.so, and vica-verca, to ensure we both avoid pulling major new dependancy chains for all users. Similarly, if at all possible, any existing libguestfs application would like to be able to 'just work' with any libvirt integration without further code changes. Now what do I think we should do. Personally I would really like to have libguestfs be integrating with the lock manager, secret manager and sVirt infrastructure in libvirt. The only real practical way for this to happen is if the appliance runs server side, spawned via the normal QEMU driver guest startup code in libvirt. So I discount any idea of running the appliance client side & tunnelling block devices over libvirt. Also note that different solutions may be required for hypervisors without libvirt. I am ignoring such hypervisors for now. If I were ignoring the requirement that libguestfs does not link to libvirt, then you could quite likely make all this happen with only a simple additional API in libvirt. We need an API to let a client open a connection to a <channel> device, using the virStreamPtr API. If the guests were not running, libguestfs would use virDomainCreate to spawn a transient, auto-detroy guest, with a custom kernel/initrd that runs the appliance, and an additional <channel> device, but with all other parts of the guest XML unchanged. This would ensure all the lock manager, sVirt and secret stuff 'just works'. If the guest is already running, libguestfs would just query the XML to find the <channel> device configuration. Then it could just use a new API like virDomainOpenChannel(virStreamPtr, const char *channelid) to get a stream to talk to the guestfs daemon with. If libguestfs were just wanting to use a random disk device, not yet associated with any guest, then it again would use virDOmainCreate to spawn a transient, auto-destroy guest, with an XML config it fully creates. This would all make it possible for any guestfs based application to work with libvirt without any app code changes. I assume it is still the case, however, that libguestfs does *not* want to link directly to libvirt and use it for guest creation. Thus my idea is to find an alternative solution that gets as close to that "ideal" setup as possible. To do this I would create what I call a bridging library, to be named 'libvirt-guestfs.so'. This would have a handful of API calls for handling the initial creation/startup of the appliance & access to the vmchannel device only, delegating everything else to normal libguestfs APIs. - int virDomainStartGuestFS(virDomainPtr dom, int flags) If the guest 'dom' is not already running, boot the guest pointing it at the libguestfs kernel/initrd + vmchannel device. - guestfs_h *virDomainOpenGuestFS(virDomainPtr dom) Server side, find the vmchannel device for the guest, open a stream for it. On the client side the stream would somehow be proxied to a UNIX domain socket. We then call the libguestfs APIs to neccessary to attach the external UNIX domain socket, create the guestfs_h handle and return it. With those two APIs (possibly 1 or 2 more), an application wanting to use an integrated libguestfs+libvirt, would use libvirt-guestfs.so to obtain their guestfs_h * handle, and then use all the other libguestfs.so APIs as normal. This avoids having to wrap every single libguestfs API in libvirt. For apps like virt-manager this would easily be workable, other existing libguestfs apps would however need to have some small changes in their initial connection setup code to optionally use libvirt-guestfs.so Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Tue, Sep 27, 2011 at 12:35:21PM +0100, Richard W.M. Jones wrote: Yeah, I think libvirt will just have to be able to run the febootstrap-supermin-helper program itself at the appropriate time. As long as we create some SELinux security policy to confine the febootstrap-supermin-helper I don't see that as a very significant problem. I think the SELinux policy would be quite straightforward to create, since the program has a pretty well defined single task to perform Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Tue, Sep 27, 2011 at 12:55:19PM +0100, Richard W.M. Jones wrote: This all sounds like a good plan to aim for. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Wed, Sep 28, 2011 at 11:14:57AM +0100, Stefan Hajnoczi wrote: This is how the Debian package of libguestfs works (Hilko's official package, not my one). However this is troublesome because it means any security problem in a dependent program is baked into the appliance. Applying a security update to the host wouldn't update this libguestfs appliance. Compare this to the way febootstrap-supermin-helper normally works (eg upstream, Fedora and RHEL): the appliance is rebuilt whenever any change is noticed in a dependent program. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top

On Wed, Sep 28, 2011 at 05:35:46PM +0100, Stefan Hajnoczi wrote: RPM & dpkg both have trigger mechanisms. The Debian package doesn't appear to use it for whatever reason. In the RPM we just don't use triggers because the checksum method we're using is more convenient and produces about the same result. We could change this but I want to look at other alternatives as well. In particular, using 9pfs might mean there's no need to explicitly build a root appliance at all (but it needs some qemu changes). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v

On Wed, Sep 28, 2011 at 06:37:17PM +0100, Daniel P. Berrange wrote: I'm fine with this. Are there security implications to allowing users to add <bootloader> clauses pointing at random scripts that get run on remote machines as different users? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top

On Wed, Sep 28, 2011 at 06:52:13PM +0100, Richard W.M. Jones wrote: Yes, but you have to consider a connection to libvirtd, to be equivalent to a root shell at this time anyway. When we get RBAC in libvirt we'll be able to control who can make such configurations, and/or whitelist bootloaders in the SELinux policy so only trusted ones can be run Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|