[libvirt-users] sharing read-only LOCAL disks among KVM guests
Hello, Is it possible to share a single host's Logical Volume among multiple local KVM guests which mount it read-only? I'm asking this because I have an idea to run multiple idential KVM guests (they all have exactly the same software installed on them), booting them from a shared local Logical Volume read-only root file system, or alternatively let them share the bulk of the software (/usr, /opt, /lib) from a common KVM host Logical volume. Is this possible? All my searches so far failed to turn up anything like this. Would it be possible using qcow2 instead of raw LV? If so - would it be worth the performance hit of switching from LV to qcow2? The motivation for this is two-fold: 1. Speed-up - I suspect that sharing the same disk block among all KVM agents (we currently have 14 KVM guests on each server) should lead to better cache utilisation and reduce disk access. Also the cost of maintaining the mutiple identical servers and keeping them up to date could be eating into our server's resource usage. 2. Maintenance - if I get to share the common disk image among all of them then I only need to install/update it once instead of once for each KVM guest (we already use Puppet but it doesn't mean it comes for free). Thanks, --Amos
On Fri, Sep 21, 2012 at 09:22:24PM +1000, Amos Shapira wrote:
Hello,
Is it possible to share a single host's Logical Volume among multiple local KVM guests which mount it read-only?
I'm asking this because I have an idea to run multiple idential KVM guests (they all have exactly the same software installed on them), booting them from a shared local Logical Volume read-only root file system, or alternatively let them share the bulk of the software (/usr, /opt, /lib) from a common KVM host Logical volume.
Is this possible? All my searches so far failed to turn up anything like this.
From the libvirt POV, there's nothing much todo except add <readonly/> inside the <disk> element. This will ensure QEMU only gets given read permission on the disk backend. The important thing is to then make sure your guests actually mount the filesystem with the readonly flag.
Would it be possible using qcow2 instead of raw LV? If so - would it be worth the performance hit of switching from LV to qcow2?
The type of backend storage doesn't really affect things if the disk is fully readonly. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 21 September 2012 21:30, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, Sep 21, 2012 at 09:22:24PM +1000, Amos Shapira wrote:
Hello,
Is it possible to share a single host's Logical Volume among multiple local KVM guests which mount it read-only?
I'm asking this because I have an idea to run multiple idential KVM guests (they all have exactly the same software installed on them), booting them from a shared local Logical Volume read-only root file system, or alternatively let them share the bulk of the software (/usr, /opt, /lib) from a common KVM host Logical volume.
Is this possible? All my searches so far failed to turn up anything like this.
From the libvirt POV, there's nothing much todo except add <readonly/> inside the <disk> element. This will ensure QEMU only gets given read permission on the disk backend. The important thing is to then make sure your guests actually mount the filesystem with the readonly flag.
Would it be possible using qcow2 instead of raw LV? If so - would it be worth the performance hit of switching from LV to qcow2?
The type of backend storage doesn't really affect things if the disk is fully readonly.
Thanks very much! That's very helpful to know. Is this something that someone has already done before (booting multiple KVM guests from shared read-only root file system) or am I on my own with this? Cheers, --Amos
On Fri, Sep 21, 2012 at 09:34:02PM +1000, Amos Shapira wrote:
On 21 September 2012 21:30, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, Sep 21, 2012 at 09:22:24PM +1000, Amos Shapira wrote:
Hello,
Is it possible to share a single host's Logical Volume among multiple local KVM guests which mount it read-only?
I'm asking this because I have an idea to run multiple idential KVM guests (they all have exactly the same software installed on them), booting them from a shared local Logical Volume read-only root file system, or alternatively let them share the bulk of the software (/usr, /opt, /lib) from a common KVM host Logical volume.
Is this possible? All my searches so far failed to turn up anything like this.
From the libvirt POV, there's nothing much todo except add <readonly/> inside the <disk> element. This will ensure QEMU only gets given read permission on the disk backend. The important thing is to then make sure your guests actually mount the filesystem with the readonly flag.
Would it be possible using qcow2 instead of raw LV? If so - would it be worth the performance hit of switching from LV to qcow2?
The type of backend storage doesn't really affect things if the disk is fully readonly.
Thanks very much! That's very helpful to know.
Is this something that someone has already done before (booting multiple KVM guests from shared read-only root file system) or am I on my own with this?
I've do it with libvirt-sandbox, but using a different approach. Instead of a shared readonly disk image, I use the plan9 filesystem to pass a chroot area from the host to the guest. Then I have to mount certain areas with tmpfs or other writable filesystems (eg /var, /tmp/, parts of /etc, etc). What you want todo is definitely doable - the hard part is just in making your OS image in a way to works with readonly root. This is typically not to well documented, so involves alot of trial and error at first. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 21 September 2012 21:40, Daniel P. Berrange <berrange@redhat.com> wrote:
On 21 September 2012 21:30, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, Sep 21, 2012 at 09:22:24PM +1000, Amos Shapira wrote:
Hello,
Is it possible to share a single host's Logical Volume among multiple local KVM guests which mount it read-only?
I'm asking this because I have an idea to run multiple idential KVM guests (they all have exactly the same software installed on them), booting
from a shared local Logical Volume read-only root file system, or alternatively let them share the bulk of the software (/usr, /opt, /lib) from a common KVM host Logical volume.
Is this possible? All my searches so far failed to turn up anything
On Fri, Sep 21, 2012 at 09:34:02PM +1000, Amos Shapira wrote: them like
this.
From the libvirt POV, there's nothing much todo except add <readonly/> inside the <disk> element. This will ensure QEMU only gets given read permission on the disk backend. The important thing is to then make sure your guests actually mount the filesystem with the readonly flag.
Would it be possible using qcow2 instead of raw LV? If so - would it be worth the performance hit of switching from LV to qcow2?
The type of backend storage doesn't really affect things if the disk is fully readonly.
Thanks very much! That's very helpful to know.
Is this something that someone has already done before (booting multiple KVM guests from shared read-only root file system) or am I on my own with this?
I've do it with libvirt-sandbox, but using a different approach. Instead of a shared readonly disk image, I use the plan9 filesystem to pass a chroot area from the host to the guest. Then I have to mount certain areas with tmpfs or other writable filesystems (eg /var, /tmp/, parts of /etc, etc).
What you want todo is definitely doable - the hard part is just in making your OS image in a way to works with readonly root. This is typically not to well documented, so involves alot of trial and error at first.
Thanks. I'll try to make my workplace interested in this and report back if/when I have interesting results. In general, I'm thinking of replicating the way that diskless NFS clients used to boot from a common root file system on an NFS server, where the only difference among them is their MAC address, which in turn lets them pull a little different information (e.g. host name) from DHCP and use this as a basis to mount the private parts of the disk (e.g. /var, /etc?) and configuration. The other question is whether I am correct in expecting that this could improve my disk IO profile, since all the guests would be accessing the same host's disk blocks. Cheers, --Amos
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|
-- [image: View my profile on LinkedIn] <http://www.linkedin.com/in/gliderflyer>
On Fri, Sep 21, 2012 at 09:48:58PM +1000, Amos Shapira wrote:
On 21 September 2012 21:40, Daniel P. Berrange <berrange@redhat.com> wrote:
On 21 September 2012 21:30, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, Sep 21, 2012 at 09:22:24PM +1000, Amos Shapira wrote:
Hello,
Is it possible to share a single host's Logical Volume among multiple local KVM guests which mount it read-only?
I'm asking this because I have an idea to run multiple idential KVM guests (they all have exactly the same software installed on them), booting
from a shared local Logical Volume read-only root file system, or alternatively let them share the bulk of the software (/usr, /opt, /lib) from a common KVM host Logical volume.
Is this possible? All my searches so far failed to turn up anything
On Fri, Sep 21, 2012 at 09:34:02PM +1000, Amos Shapira wrote: them like
this.
From the libvirt POV, there's nothing much todo except add <readonly/> inside the <disk> element. This will ensure QEMU only gets given read permission on the disk backend. The important thing is to then make sure your guests actually mount the filesystem with the readonly flag.
Would it be possible using qcow2 instead of raw LV? If so - would it be worth the performance hit of switching from LV to qcow2?
The type of backend storage doesn't really affect things if the disk is fully readonly.
Thanks very much! That's very helpful to know.
Is this something that someone has already done before (booting multiple KVM guests from shared read-only root file system) or am I on my own with this?
I've do it with libvirt-sandbox, but using a different approach. Instead of a shared readonly disk image, I use the plan9 filesystem to pass a chroot area from the host to the guest. Then I have to mount certain areas with tmpfs or other writable filesystems (eg /var, /tmp/, parts of /etc, etc).
What you want todo is definitely doable - the hard part is just in making your OS image in a way to works with readonly root. This is typically not to well documented, so involves alot of trial and error at first.
Thanks. I'll try to make my workplace interested in this and report back if/when I have interesting results.
In general, I'm thinking of replicating the way that diskless NFS clients used to boot from a common root file system on an NFS server, where the only difference among them is their MAC address, which in turn lets them pull a little different information (e.g. host name) from DHCP and use this as a basis to mount the private parts of the disk (e.g. /var, /etc?) and configuration.
The other question is whether I am correct in expecting that this could improve my disk IO profile, since all the guests would be accessing the same host's disk blocks.
The normal KVM recommended practice is to use cache=none and let the guest OS cache I/O on the basis that you can then directly control guest memory usage. This is probably the exact opposite of what you want though. To maximise performance I expect you want to let the host OS aggressively cache the shared disk image (cache=writethrough/unsafe), and limit (but not eliminate) the amount of extra caching the guest does Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Fri, Sep 21, 2012 at 09:22:24PM +1000, Amos Shapira wrote:
2. Maintenance - if I get to share the common disk image among all of them then I only need to install/update it once instead of once for each KVM guest (we already use Puppet but it doesn't mean it comes for free).
How do you plan on upgrading the read-only VMs after the initial installation, or are they short-lived and frequently recreated from scratch using a fresh image? Read-only, shared /usr between machines is a nice idea in theory, but at least the modern linux distros' package management tend not to support it that well in practice. IMHO using a QCOW2 backing files as base templates works pretty well and gives you similar disk space savings, at least initially. And KSM does the same with RAM.
On 21 September 2012 21:58, Henrik Ahlgren <pablo@seestieto.com> wrote:
2. Maintenance - if I get to share the common disk image among all of
On Fri, Sep 21, 2012 at 09:22:24PM +1000, Amos Shapira wrote: them
then I only need to install/update it once instead of once for each KVM guest (we already use Puppet but it doesn't mean it comes for free).
How do you plan on upgrading the read-only VMs after the initial installation, or are they short-lived and frequently recreated from scratch using a fresh image? Read-only, shared /usr between machines is a nice idea in theory, but at least the modern linux distros' package management tend not to support it that well in practice.
The KVM agents host Atlassian Bamboo build agents ( http://www.atlassian.com/software/bamboo/overview). I want them to run continuously and be ready to be used for builds but: 1. I can take them down for maintenance any time between builds. 2. They are disposable (i.e. loosing them for disk crash is NOT a catastrophe), so I want maximum speed even on expense of less (or actually no) redundancy. The only way to update their software that I could think of so far is to rebuild the base image on the KVM host and reboot the KVM guests from the new image next time they become idle. Have you got other ideas?
IMHO using a QCOW2 backing files as base templates works pretty well and gives you similar disk space savings, at least initially. And KSM does the same with RAM.
Thanks for the suggestion. I am not interested in saving disk space, I have enough of it. The main motivation for this exercise the potential speed-up from having some disk blocks read once then shared by all the KVM agents in memory. About KSM - my understanding is that it doesn't come free (the KVM's have to madvise(2) about candidate memory pages, and the kernel still has to scan the memory to find them) and I assume that I can remove the need for KSM since I can already tell that the disk images are identical, so why let the kernel try to find the identical pages? Cheers, --Amos -- [image: View my profile on LinkedIn] <http://www.linkedin.com/in/gliderflyer>
participants (3)
-
Amos Shapira -
Daniel P. Berrange -
Henrik Ahlgren