2015-03-02 16:59 GMT+03:00 Daniel P. Berrange <berrange@redhat.com>:

Conceptually creating a target for the overlay is easy, the difficult question is how do we expect apps to actually manage the overlays once created. In particular at which point do they get deleted. I don't think we would want delete them at container shutdown, at least not for persistent containers - it could work for transient containers. At the same time we would definitely not want to do it during undefine either since that is really intended to only remove the config. Deleting the data could take a non-negligble amount of time to complete if the container had made lots of changes to its filesystem vs the backing store. So this all suggests and out of band mechanism for overlayfs management.

So It feels like we might need to integrate the overlayfs support into the storage pools implementation instead. eg in the filesystem storage pool driver, we could allow for the creation and deletion of overlays for directories. Then the LXC driver would not need any changes at all -it would simply point an a directory that's an overlay and use the existing type=mount support.

So as i understand i need to add overlayfs like virStorageBackendFileSystem for example virStorageBackendOvlFileSystem but i don't understand how can this pool be used in case of many containers. May be i misunderstand something? -- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru jabber: vase@selfip.ru

On Mon, Mar 02, 2015 at 07:45:18PM +0400, Vasiliy Tolstov wrote:

2015-03-02 18:22 GMT+03:00 Vasiliy Tolstov <v.tolstov@selfip.ru>:

...

So as i understand i need to add overlayfs like virStorageBackendFileSystem for example virStorageBackendOvlFileSystem but i don't understand how can this pool be used in case of many containers. May be i misunderstand something?

Or you mean to add .conf/storage_conf.c overlay type fs and add logic to processing this. And for such pools i can create xml like: <pool type="fs"> <name>overlay</name> <source> <dir path="/var/lib/libvirt/filesystem"/> <format type='overlay'/> </source> <target> <path>/var/lib/libvirt/overlay</path> </target> </pool>

Ok, I've read a little bit more about overlayfs and see that my suggestion was incorrect, so ignore everything I said originally. In particular, I was mistakenly coupling the action of mounting/unmounting the overlay with the action of creating/deleting the directory containing the overlay. These are two completely independant actions and so don't need to be coupled in any way. Only the creating/deleting directory bit makes sense in the context of storage pools and that code would not in fact be at all specific to overlayfs. So lets ignore that and just consider the mount/unmount part of the problem. This we can do just in the context of the LXC guest XML config. Currently LXC supports the following interesting filesystem types 1. External bind mounts - mount dir in host to dir in containre <filesystem type='mount' accessmode='passthrough'> <source dir='/path/on/host'/> <target dir='/path/in/container'/> </filesystem> 2. Internal bind mounts - mount dir in container to other dir in container <filesystem type='bind' accessmode='passthrough'> <source dir='/path/in/container'/> <target dir='/other/path/in/container'/> </filesystem> 3. Block mounts - mount block device in host to dir in container <filesystem type='block' accessmode='passthrough'> <source file='/dev/sdb1'/> <target dir='/path/in/container'/> </filesystem> 4. File mounts - mount image file in host to dir in container <filesystem type='file' accessmode='passthrough'> <driver name='nbd' type='qcow2'/> <source file='/file/on/host/guest.img'/> <target dir='/path/in/container'/> </filesystem> Reading the kernel docs at: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documen... For overlayfs to be used, we have to represent 4 pieces of info in total 1. one or more lower directory paths 2. one upper directory path 3. one work directory path (which must be empty) 4. the target mount directory mount -t overlay overlay -olowerdir=/lower1:/lower1:/lower3,\ upperdir=/upper,workdir=/work /merged The fourth item obvious maps to the <target> element in the XML, which leave us 3 extra directory parameters to be added to the XML in some manner. For added fun^H^H^Hpain, any of those lower dirs can themselves be overlayfs merged mount locations. IOW we have to consider the possibility of arbitrarily deep nesting too. We could try to fit in extra parameters in each of the existing filesystem types to support setting up of an overlay. Or we could just define a new overlayfs type, or we could leverage the <driver> type field somehow. For disks, we have introduced the idea of nested backing stores. So a qcow2 file, pointing to a qcow2 file pointing to an LVM device would end up looking like: <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/domain.qcow'/> <backingStore type='file'> <format type='qcow2'/> <source file='/var/lib/libvirt/images/snapshot.qcow'/> <backingStore type='block'> <format type='raw'/> <source dev='/dev/mapper/base'/> <backingStore/> </backingStore> </backingStore> <target dev='vdd' bus='virtio'/> </disk> I think it could be appropriate to follow that kind of approach here too, which would let us in fact deal with the various different types of mounts we have at each level. On balance I think we probably want to introduce a new top level filesystem type for overlayfs. This would have 2 source dirs, the first would be the upper dir, the second would be the work dir. There would be backing stores for the lower dirs So for the simplest operation # mount -t overlay overlay -olowerdir=/lower1:/lower1:/lower3,\ upperdir=/upper,workdir=/work /merged <filesystem type='overlay' accessmode='passthrough'> <source dir='/upper'/> <source dir='/work'/> <backingStore type='dir'> <target dir='/lower1'/> </backingStore> <backingStore type='dir'> <target dir='/lower2'/> </backingStore> <backingStore type='dir'> <target dir='/lower3'/> </backingStore> <target dir='/merged'/> </filesystem> That would be a flexible starting point for a first implementation. If we later want to get more adventurous and later consider adding further kinds of backing store. For example, if we want nested overlays # mount -t overlay overlay -olowerdir=/backing/lower1:/backing/lower2,\ upperdir=/backing/upper,workdir=/backing/work /backing/merged # mount -t overlay overlay -olowerdir=/backing/merged:/lower1:/lower3,\ upperdir=/upper,workdir=/work /merged <filesystem type='overlay' accessmode='passthrough'> <source dir='/upper'/> <source dir='/work'/> <backingStore type='overlay'> <source dir='/backing/upper'/> <source dir='/backing/work'/> <backingStore type='dir'> <target dir='/backing/lower1'/> </backingStore> <backingStore type='dir'> <target dir='/backing/lower2'/> </backingStore> <target dir='/backing/merged'/> </backingStore> <backingStore type='dir'> <target dir='/lower2'/> </backingStore> <backingStore type='dir'> <target dir='/lower3'/> </backingStore> <target dir='/merged'/> </filesystem> And if we want to get really adventurous and use qcow2 image as a backing # qemu-nbd /guest.img # mount /dev/nbd0 /lower1 # mount -t overlay overlay -olowerdir=/lower1,\ upperdir=/upper,workdir=/work /merged <filesystem type='overlay' accessmode='passthrough'> <source dir='/upper'/> <source dir='/work'/> <backingStore type='file'> <driver name='nbd' type='qcow2'/> <source file='/guest.img'/> <target dir='/lower1'/> </backingStore> <target dir='/merged'/> </filesystem> It is perfectly fine if you just implement the first simple case, with just a single overlayfs and no nesting. The key is just that we get this XML design right to allow the future extensions I describe Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|