7:38 a.m.
Hello. I'm interesting in implementing overlayfs support to lxc libvirt driver.
As i see i simply can utilize filesystem type='template' for this usage.
In case os template fs i need to mount with lowerdir
/var/lib/libvirt/filesystems/template to
/var/lib/libvirt/filesystems/name
But i have problem: overlayfs fs need upperdir that store differences
and mountpoint that used by clients. In case of libvirt usage -
upperdir need to be places in /var/lib/libvirt/filesystems, but where
i can put mountpoint dir ? /var/run/libvirt or in /tmp/libvirt-XXXXX ?
Also what is the preferred way to integrate overlayfs support to
libvirt? I can addd mount type to filesystem type='mount' or use (like
now) template type. What is the preferred way?
Thanks
--
Vasiliy Tolstov,
e-mail: v.tolstov(a)selfip.ru
jabber: vase(a)selfip.ru
7:59 a.m.
On Mon, Mar 02, 2015 at 05:38:11PM +0400, Vasiliy Tolstov wrote:
Hello. I'm interesting in implementing overlayfs support to lxc
libvirt driver.
As i see i simply can utilize filesystem type='template' for this usage.
The semantics of the 'template' type are that the virt driver unpacks a
pre-built named template image and uses that for the guest filesystem.
I don't think we should try to twist that into use for overlayfs. In
any case, all of the existing mount types could conceivably benefit
from the use of an overlay.
In case os template fs i need to mount with lowerdir
/var/lib/libvirt/filesystems/template to
/var/lib/libvirt/filesystems/name
But i have problem: overlayfs fs need upperdir that store differences
and mountpoint that used by clients. In case of libvirt usage -
upperdir need to be places in /var/lib/libvirt/filesystems, but where
i can put mountpoint dir ? /var/run/libvirt or in /tmp/libvirt-XXXXX ?
Also what is the preferred way to integrate overlayfs support to
libvirt? I can addd mount type to filesystem type='mount' or use (like
now) template type. What is the preferred way?
Thanks
Conceptually creating a target for the overlay is easy, the difficult
question is how do we expect apps to actually manage the overlays once
created. In particular at which point do they get deleted. I don't
think we would want delete them at container shutdown, at least not
for persistent containers - it could work for transient containers.
At the same time we would definitely not want to do it during undefine
either since that is really intended to only remove the config. Deleting
the data could take a non-negligble amount of time to complete if the
container had made lots of changes to its filesystem vs the backing
store. So this all suggests and out of band mechanism for overlayfs
management.
So It feels like we might need to integrate the overlayfs support into
the storage pools implementation instead. eg in the filesystem storage
pool driver, we could allow for the creation and deletion of overlays for
directories. Then the LXC driver would not need any changes at all -it
would simply point an a directory that's an overlay and use the existing
type=mount support.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
9:22 a.m.
2015-03-02 16:59 GMT+03:00 Daniel P. Berrange <berrange(a)redhat.com>:
Conceptually creating a target for the overlay is easy, the
difficult
question is how do we expect apps to actually manage the overlays once
created. In particular at which point do they get deleted. I don't
think we would want delete them at container shutdown, at least not
for persistent containers - it could work for transient containers.
At the same time we would definitely not want to do it during undefine
either since that is really intended to only remove the config. Deleting
the data could take a non-negligble amount of time to complete if the
container had made lots of changes to its filesystem vs the backing
store. So this all suggests and out of band mechanism for overlayfs
management.
So It feels like we might need to integrate the overlayfs support into
the storage pools implementation instead. eg in the filesystem storage
pool driver, we could allow for the creation and deletion of overlays for
directories. Then the LXC driver would not need any changes at all -it
would simply point an a directory that's an overlay and use the existing
type=mount support.
So as i understand i need to add overlayfs like
virStorageBackendFileSystem for example virStorageBackendOvlFileSystem
but i don't understand how can this pool be used in case of many containers.
May be i misunderstand something?
--
Vasiliy Tolstov,
e-mail: v.tolstov(a)selfip.ru
jabber: vase(a)selfip.ru
9:45 a.m.
2015-03-02 18:22 GMT+03:00 Vasiliy Tolstov <v.tolstov(a)selfip.ru>:
So as i understand i need to add overlayfs like
virStorageBackendFileSystem for example virStorageBackendOvlFileSystem
but i don't understand how can this pool be used in case of many containers.
May be i misunderstand something?
Or you mean to add .conf/storage_conf.c overlay type fs and add logic
to processing this. And for such pools i can create xml like:
<pool type="fs">
<name>overlay</name>
<source>
<dir path="/var/lib/libvirt/filesystem"/>
<format type='overlay'/>
</source>
<target>
<path>/var/lib/libvirt/overlay</path>
</target>
</pool>
--
Vasiliy Tolstov,
e-mail: v.tolstov(a)selfip.ru
jabber: vase(a)selfip.ru
5:20 a.m.
On Mon, Mar 02, 2015 at 07:45:18PM +0400, Vasiliy Tolstov wrote:
2015-03-02 18:22 GMT+03:00 Vasiliy Tolstov
<v.tolstov(a)selfip.ru>:
> So as i understand i need to add overlayfs like
> virStorageBackendFileSystem for example virStorageBackendOvlFileSystem
> but i don't understand how can this pool be used in case of many containers.
> May be i misunderstand something?
Or you mean to add .conf/storage_conf.c overlay type fs and add logic
to processing this. And for such pools i can create xml like:
<pool type="fs">
<name>overlay</name>
<source>
<dir path="/var/lib/libvirt/filesystem"/>
<format type='overlay'/>
</source>
<target>
<path>/var/lib/libvirt/overlay</path>
</target>
</pool>
Ok, I've read a little bit more about overlayfs and see that my
suggestion was incorrect, so ignore everything I said originally.
In particular, I was mistakenly coupling the action of mounting/unmounting
the overlay with the action of creating/deleting the directory containing
the overlay. These are two completely independant actions and so don't
need to be coupled in any way. Only the creating/deleting directory bit
makes sense in the context of storage pools and that code would not in
fact be at all specific to overlayfs. So lets ignore that and just
consider the mount/unmount part of the problem. This we can do just in
the context of the LXC guest XML config.
Currently LXC supports the following interesting filesystem types
1. External bind mounts - mount dir in host to dir in containre
<filesystem type='mount' accessmode='passthrough'>
<source dir='/path/on/host'/>
<target dir='/path/in/container'/>
</filesystem>
2. Internal bind mounts - mount dir in container to other dir in container
<filesystem type='bind' accessmode='passthrough'>
<source dir='/path/in/container'/>
<target dir='/other/path/in/container'/>
</filesystem>
3. Block mounts - mount block device in host to dir in container
<filesystem type='block' accessmode='passthrough'>
<source file='/dev/sdb1'/>
<target dir='/path/in/container'/>
</filesystem>
4. File mounts - mount image file in host to dir in container
<filesystem type='file' accessmode='passthrough'>
<driver name='nbd' type='qcow2'/>
<source file='/file/on/host/guest.img'/>
<target dir='/path/in/container'/>
</filesystem>
Reading the kernel docs at:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Docu...
For overlayfs to be used, we have to represent 4 pieces of
info in total
1. one or more lower directory paths
2. one upper directory path
3. one work directory path (which must be empty)
4. the target mount directory
mount -t overlay overlay -olowerdir=/lower1:/lower1:/lower3,\
upperdir=/upper,workdir=/work /merged
The fourth item obvious maps to the <target> element in the XML,
which leave us 3 extra directory parameters to be added to the
XML in some manner.
For added fun^H^H^Hpain, any of those lower dirs can themselves
be overlayfs merged mount locations. IOW we have to consider the
possibility of arbitrarily deep nesting too.
We could try to fit in extra parameters in each of the existing
filesystem types to support setting up of an overlay. Or we could
just define a new overlayfs type, or we could leverage the <driver>
type field somehow.
For disks, we have introduced the idea of nested backing stores.
So a qcow2 file, pointing to a qcow2 file pointing to an LVM
device would end up looking like:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/domain.qcow'/>
<backingStore type='file'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/snapshot.qcow'/>
<backingStore type='block'>
<format type='raw'/>
<source dev='/dev/mapper/base'/>
<backingStore/>
</backingStore>
</backingStore>
<target dev='vdd' bus='virtio'/>
</disk>
I think it could be appropriate to follow that kind of approach
here too, which would let us in fact deal with the various different
types of mounts we have at each level.
On balance I think we probably want to introduce a new top level
filesystem type for overlayfs. This would have 2 source dirs,
the first would be the upper dir, the second would be the work
dir. There would be backing stores for the lower dirs
So for the simplest operation
# mount -t overlay overlay -olowerdir=/lower1:/lower1:/lower3,\
upperdir=/upper,workdir=/work /merged
<filesystem type='overlay' accessmode='passthrough'>
<source dir='/upper'/>
<source dir='/work'/>
<backingStore type='dir'>
<target dir='/lower1'/>
</backingStore>
<backingStore type='dir'>
<target dir='/lower2'/>
</backingStore>
<backingStore type='dir'>
<target dir='/lower3'/>
</backingStore>
<target dir='/merged'/>
</filesystem>
That would be a flexible starting point for a first implementation.
If we later want to get more adventurous and later consider adding
further kinds of backing store. For example, if we want nested
overlays
# mount -t overlay overlay -olowerdir=/backing/lower1:/backing/lower2,\
upperdir=/backing/upper,workdir=/backing/work /backing/merged
# mount -t overlay overlay -olowerdir=/backing/merged:/lower1:/lower3,\
upperdir=/upper,workdir=/work /merged
<filesystem type='overlay' accessmode='passthrough'>
<source dir='/upper'/>
<source dir='/work'/>
<backingStore type='overlay'>
<source dir='/backing/upper'/>
<source dir='/backing/work'/>
<backingStore type='dir'>
<target dir='/backing/lower1'/>
</backingStore>
<backingStore type='dir'>
<target dir='/backing/lower2'/>
</backingStore>
<target dir='/backing/merged'/>
</backingStore>
<backingStore type='dir'>
<target dir='/lower2'/>
</backingStore>
<backingStore type='dir'>
<target dir='/lower3'/>
</backingStore>
<target dir='/merged'/>
</filesystem>
And if we want to get really adventurous and use qcow2 image as a
backing
# qemu-nbd /guest.img
# mount /dev/nbd0 /lower1
# mount -t overlay overlay -olowerdir=/lower1,\
upperdir=/upper,workdir=/work /merged
<filesystem type='overlay' accessmode='passthrough'>
<source dir='/upper'/>
<source dir='/work'/>
<backingStore type='file'>
<driver name='nbd' type='qcow2'/>
<source file='/guest.img'/>
<target dir='/lower1'/>
</backingStore>
<target dir='/merged'/>
</filesystem>
It is perfectly fine if you just implement the first simple case, with
just a single overlayfs and no nesting. The key is just that we get
this XML design right to allow the future extensions I describe
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
3554
days inactive
3556
days old
4 comments
2 participants
participants (2)
-
Daniel P. Berrange -
Vasiliy Tolstov