Daniel P. Berrange wrote: Since that thread is split across two months, can I bring to everyone's attention the post I made yesterday: http://www.redhat.com/archives/libvir-list/2007-October/msg00057.html In particular the concept at the end that we shouldn't even try to support every possible remote storage, but instead allow the administrator to write "scriptlets" (small shell scripts with a well-defined input & output) to perform a set of operations: ----- /etc/libvirtd.conf ------------------- allocate partition: "lvcreate -L %size -n %name XenVolGroup" list partitions: "lvs --xml" ---------- We can provide sample scriptlets for different operating systems and storage configurations. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903

Daniel P. Berrange wrote: Well, a basic set of operations would be whatever we need to implement virt-install/virt-manager remotely now, plus other suggestions as they come along. From a fairly brief scan of the virt-install & virt-manager code that would be: - Create an empty a file with given name & size & sparseness. - Detect if a named device or file exists (basically a remote stat). - Copy image to remote temporary file (for kernel/CD-ROM). - Check free space (remote statvfs). It might be nice to list LVs, but it doesn't seem to be necessary to implement remote virt-* at the moment (AFAICS). Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903

Daniel P. Berrange wrote: Yeah, I think that's the right place to start. As you say, there are just too many underlying storage technologies to go after all of them at once. <snip> Kind of, though I think there are actually two concepts of Volumes here (if I am understanding correctly). The first concept of volume is "raw storage" -> what you assign to a pool. The second concept is "Volume exported for a guest". I'm not sure that we want to Nest those concepts. The question I have with all of this is whether it really belongs in libvirt at all. Many of these concepts apply to bare-metal provisioning as well; so it might be a good idea to have a separate "libstorage" that libvirt links to, and that other tools might use. The problem with most of these, as we all know, is that they only have command-line utilities, and no corresponding libraries. That makes it difficult for a library like libvirt to support them. That is, we can shell out to the commands, but then we run into a situation where different versions of the LVM command, for example, have different output. We have now effectively tied ourselves to a particular version of a tool, which is fairly disappointing. Also, as we have seen with xend, spawning external tools to do work makes error reporting far more difficult (maybe impossible). So that leaves us the tough question of what to do here. Ideally we would abstract all of the above tools into libraries, and re-write the tools to use those, and then have libvirt use the same. I'm not sure if it is practical, however, to wait that long to do such things. What I might suggest is a hybrid approach. Do the initial implementation with what we have (namely the command line utilities, possibly utilizing rjones "scriplet" concept). In parallel, make sure someone starts on making real libraries of the tools, so that we can benefit from that later on. Yep. As long as we make sure the XML is flexible enough to handle the remaining stuff, this seems like a good first place to start. Chris Lalancette

Daniel P. Berrange wrote: That's true. I've always viewed the storage allocated to a guest, whether it be a partition, a file, etc., as an opaque object that the guest can do anything it wants with, even if the host doesn't understand it. There is an argument to be made for the host being able to do maintenance on guest disks (if, for some reason, you can't boot the guest); but I honestly think this sort of maintenance is best done inside the guest container (via the "normal" guest rescue modes - LiveCD, Knoppix, Windows Rescue, etc). Agreed that the scope for a libstorage could get out of hand. But I think if we keep the initial scope targetted enough for what we want (i.e. integration with libvirt/virt matters), we can get away with having a separate library, and the rest can grow organically from that. Really, the only part that needs to be generic is the XML; if we can get that down, the implementation doesn't matter. Basically with this model we wouldn't be dependent on libvirt internal data structures/functions, so we wouldn't have the pain of extracting it later. Of course, that is more work too :). Chris Lalancette

On Wed, Oct 17, 2007 at 02:55:21PM +0100, Mark McLoughlin wrote: This mention of recursion seems to have caused alot of confusion... All I really mean by it is that libvirt has two notions - A volume - A pool When you define a pool, the XML description may refer to one of more volumes which are the source of the pool. eg if you define a new LVM volume group, you provide one or more physical volumes. Given a pool you may carve out out or more volumes. eg you carve out logical volumes. So, the APIs from a libvirt level aren't directly 'recursive' - you just have a concept of a pool & a volume object. As you work with these two concepts you may end up creating things which are recursive in nature. In fact even if you don't conciously define anything recursive, it is indirectly recursive, since a Fedora guest will turn a disk it is assigned into a LVM vol group & logical vols. So in summary, the 'recursion' is just a fundamental property of the storage stack, but not something we need to directly express in libvirt APIs - the mere concepts of a volume & a pool is sufficient. It depends on deployment scenario, but certainly in a 'fat dom0' scenario I imagine you couldd always provide a default pool (eg /var/lib/xen/images) Whether to treat the host as pool for its physically attached devices is interesting idea. One alternative is to have an explicit API for listing all host devices (eg, 'lshal'), since I'd certainly like to be able to enumerate any USB, devices & any PCI devices, as well as any physical network adapters. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|

On Tue, Oct 16, 2007 at 04:19:29PM +0100, Daniel P. Berrange wrote: It was mentioned offlist that I didn't include security/authorization in this mail. I had it in my offline notes... - NFS - server side ACL based on client IP ranges - Kerberos GSSAPI. Client credentials taken from /etc/krb5.tab - iSCSI - server side ACL based on client IP ranges - CHAP username+password supplied when attaching target to client - Spec for Kerberos. Not GSSAPI based. Not implemented in Linux client or server. Frowned upon by IETF kerberos experts since it isn't GSSAPI - QCow - passphrase needed by process (eg QEMU) accessing the file - dm-crypt - passphrase needed when activating the volume Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|

Here is an updated document about storage which attempts to formulate the various pieces of metadata in an XML representation. In addition I have introduced a 3rd concept of a 'device'. A device represents any physical device attached ato a host, be it a disk, a sound card, a USB gizmo, or anything else you would see with 'lshal' (or in sysfs). I considered Mark's suggestion that we have a 'host pool' in which physical storage devices live, but I think it is important to directly represent the physical devices as a concept separately from pools & volumes. This is because we need this in other areas - we we need network device info when setting up networking in guests & virtual networks. We need USB & PCI device info todo device pass-through to the guest, etc. Finally also included more info about permissions & security. Taxonomy of storage types ========================= | +- Block | | | +- Disk | | | | | +- Direct attached | | | | | | | +- IDE/ATA disk | | | +- SCSI disk | | | +- FibreChannel disk | | | +- USB disk/flash | | | +- FireWire disk/flash | | | | | +- Remote attached | | | | | +- iSCSI disk | | +- GNBD disk | | | +- Partition | | | +- Virtual | | | +- Direct attached | | | | | +- LVM | | +- ZFS | | | +- Remote attached | | | +- Cluster LVM | +- FileSystem | | | +- Directed attached | | | | | +- ext2/3/4 | | +- xfs | | +- ZFS | | | +- Remote attached | | | +- NFS | +- GFS | +- OCFS2 | +- Directory | +- File | +- Raw allocated +- Raw sparse +- QCow2 +- VMDK Storage attributes ================== - Local vs network (ext3 vs NFS, SCSI vs iSCSI) - Private vs shared (IDE vs FibreChannel) - Pool vs volume (LVM VG vs LV, Directory vs File, Disk vs Partition) - Container vs guest (OpenVZ vs Xen) - Attributes - Compressed - Encrypted - Auto-extend - Snapshots - RO - RW - Partition table - MBR - GPT - UUID - 16 hex digits - Unique string - SCSI WWID (world wide ID) - Local Path(s) - Server Hostname - Server Identifier (export path/target) - MAC security label (SELinux) - Redundancy (mirrored/striped/multipath) - Pool operation - RO - RW - Authentication - Username / Password - Client IP/MAC address - Kerberos / GSSAPI - Passphrase Nesting hierarchy ================= - 1 x Host -> N x iSCSI target -> N x LUN -> N x Partition - N x Disk/Partition -> 1 x LVM VG -> B x LVM LV - 1 x Filesystem -> N x directory -> N x file - 1 x File -> 1 x Block (loopback) Application users ================= - virt-manager / virt-install - Enumerate available pools - Allocate volume from pool - Create guest with volume - virt-clone - Copy disks - Snapshot disks - virt-df - Filesystem usage - pygrub - Extract kernel/initrd from filesystem - virt-factory - Manage storage pools - Pre-migration sanity checks - virt-backup - Snapshot disks - virt-p2v - Snapshot disks Storage representation ====================== Three core concepts - Device - a physical device attached to a host - associated with a bus / subsystem (scsi,usb,ide,etc) - bus specific identifier (vendor+product ID?) - a driver type - unique id / serial number - device name for its current mapping into the filesystem - Pool - a pool of storage - contains free space - allocate to provide volumes - compromised of devices, or a remote server - Volume - a chunk of storage - assignable to a guest - part of a pool XML description =============== Storage pools ------------- High level - Type - the representation of the storage pool - Source - the underlying data storage location - Target - mapping to local filesystem (if applicable) The XML only provides information that describes the pool itself. ie, information about phyiscal devices underlying the pool is not not maintained here. - A directory within a filesystem <pool type="dir"> <name="xenimages"/> <uuid>12345678-1234-1234-1234-123456781234</uuid> <target file="/var/lib/xen/images"/> <permissions> <mode>0700</mode> <owner>root</owner> <group>virt</group> <label>xen_image_t</label> </permissions> </pool> - A dedicated filesystem <pool type="fs"> <name="xenimages"/> <uuid>12345678-1234-1234-1234-123456781234</uuid> <source dev="/dev/sda1"/> <target file="/var/lib/xen/images"/> <permissions> <mode>0700</mode> <owner>root</owner> <group>virt</group> <label>xen_image_t</label> </permissions> </pool> - A dedicated disk <pool type="disk"> <name="xenimages"/> <uuid>12345678-1234-1234-1234-123456781234</uuid> <source dev="/dev/sda"/> <permissions> <mode>0700</mode> <owner>root</owner> <group>virt</group> <label>xen_image_t</label> </permissions> </pool> - A logical volume group with 3 physical volumes <pool type="lvm"> <name="xenimages"/> <uuid>12345678-1234-1234-1234-123456781234</uuid> <source dev="/dev/sda1"/> <source dev="/dev/sdb1"/> <source dev="/dev/sdc1"/> <target dev="/dev/VirtVG"/> </pool> - A network filesystem <pool type="nfs"> <name="xenimages"/> <uuid>12345678-1234-1234-1234-123456781234</uuid> <source host="someserver" export="/vol/files"> <auth type="kerberos" keytab="/etc/server.tab"/> </source> <target file="/var/lib/xen/images"/> <permissions> <mode>0700</mode> <owner>root</owner> <group>virt</group> <label>xen_image_t</label> </permissions> </pool> - An iSCSI target <pool type="iscsi"> <name="xenimages"/> <uuid>12345678-1234-1234-1234-123456781234</uuid> <source host="someserver" export="sometarget"> <auth type="chap" username="joe" password="123456"/> </source> </pool> XXX Some kind of indiciation as to whether a pool allows creation of new volumes, or merely use of existing ones XXX flag for whether volumes will be file or block based XXX capacity / usage information if available XXX indicate whether pool can be activated/deactived, vs permanently in active state Storage volumes --------------- High level - Unique name within pool - Data format type (qcow, raw, vmdk) - FS / Dir / NFS volume <volume type="file"> <name>foo</name> <format type="qcow2"> <encrypt key="123456"/> <compress/> </format> <capacity>1000000</capacity> <allocation>100</allocation> <permissions> <mode>0700</mode> <owner>root</owner> <group>virt</group> <label>xen_image_t</label> </permissions> <target file="/var/lib/xen/images/foo.img"/> </volume> - iSCSI / LVM / Partition <volume type="block"> <name>foo</name> <capacity>1000000</capacity> <allocation>100</allocation> <permissions> <mode>0700</mode> <owner>root</owner> <group>virt</group> <label>xen_image_t</label> </permissions> <target dev="/dev/HostVG/foo"/> <snapshots> <snapshot name="bar"/> </snapshots> <volume> XXX VMWare's VMDK can be made up of many chained files XXX QCow stores snapshots internally, with a name, while LVM stores them as separate volumes with a link. Listing snapshots along side master volume seems allows both to be represented. XXX flag to indiciate whether it is resizeable ? Host devices ------------ This is not just limited to storage devices. Basically a representation of the same data provided by HAL (cf lshal) - Opaque name, vendor & product strings - Subsystem specific unique identifier for vendor/product/model - Capability type, eg storage, sound, network, etc <device> <name>/org/freedesktop/Hal/devices/volume_part2_size_99920701440</name> <vendor name="Some Vendor"/> <product name="Some Disk"/> <subsystem type="usb"> <product id="2345"/> <vendor id="2345"/> </subsystem> <class type="storage"> <block dev="/dev/ <bus type="ide"/> <drive type="cdrom"/> </capability> </device> NB, 'class' sort of maps to HAL's 'capability' field. Though HAL allows for multiple capabilities per device. Operations ========== Limited set of operations to perform For devices: - List devices - List devices by class - Lookup by path - Lookup by name For pools: - List pools (logical volume groups, partitioned devs, filesystems) - Define pool (eg create directory, or define iSCSI target) - Undefine pool - Activate pool (mount NFS volume, login to iSCSI target) - Deactivate pool - Dump pool XML - Lookup by path - Lookup by UUID - Lookup by name For volumes - List volumes (takes a pool as a param) - Create volume (takes a pool as a param) - Destroy volume - Resize volume - Copy volume - Snapshot volume - Dump volume XML - Lookup by path - Lookup by UUID - Lookup by name http://www.redhat.com/archives/libvir-list/2007-February/msg00010.html http://www.redhat.com/archives/libvir-list/2007-September/msg00119.html Implementation ============== - devices - sysfs / HAL - FileSystem/Directory/File - POSIX APIs - LVM - lvm tools - Disk/partitions - sysfs / HAL / parted - iSCSI - sysfs / HAL / iscsi utils - ZFS - ZFS tools NB, HAL gets all its info from sysfs. So can choose to use HAL, or go directly to sysfs. The former is more easily portable to Solaris, but does required the software dependancy stack to include HAL, DBus, ConsoleKit, PolicyKit and GLib. Already have DBus dep via Avahi. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|