On Fri, 2007-02-09 at 16:18 -0500, Hugh Brock wrote: * Investigate Conga's cluster and non-cluster remotely-accessible LVM management, which sounds like it would fit the bill? APIs are all XMLRPC, IIRC, so they're extensible and flexible. http://sourceware.org/cluster/conga/ -- Lon

Lon Hohberger wrote: It does general partition and file system creation tasks as well. The next release will even help you set up iscsi initiator and target. -j

On Mon, 2007-02-12 at 11:23 -0800, David Lutterkort wrote: Hrm, I thought it was on sources.redhat.com... Jim? -- Lon

Lon Hohberger wrote: It is on sources.redhat.com/cluster/conga

On Mon, 2007-02-12 at 17:53 +0000, Daniel P. Berrange wrote: Thanks for the clarification! I'm comfortable with that explanation. My initial response was without a full understanding of the context & requirements. -- Lon

Hugh Brock <hbrock(a)redhat.com&gt; wrote: Hi Hugh, I am indeed looking at GNU parted. Currently just in getting-to-know-you mode, but a few of my patches have already gone in. FYI, here are some of the relevant resources: Sources: Upstream source repository: http://git.debian.org/?p=parted/parted.git;a=summary Check out sources on the trunk via: cg-clone http://git.debian.org/git/parted/parted.git Bug tracking and development plans: http://parted.alioth.debian.org/cgi-bin/trac.cgi http://parted.alioth.debian.org/cgi-bin/trac.cgi/wiki/PlanningEdge Mailing lists: bug-parted: subscribe: http://mail.gnu.org/mailman/listinfo/bug-parted read: http://news.gmane.org/gmane.comp.gnu.parted.bugs parted-devel http://lists.alioth.debian.org/mailman/listinfo/parted-devel parted-edge (discuss next major version, 2.0): http://lists.alioth.debian.org/mailman/listinfo/parted-edge IRC: #parted on irc.freenode.net

Jim Meyering wrote: For my point of view, as former sysadmin, virtualisation and LVM are such a natural fit for each other that I can hardly imagine _not_ provisioning new virtual servers from space in a VG. I did look at the API for libparted a few months ago (actually from the rather ancient released version on gnu.org) and it didn't look to me like there was any way to express LVM notions through the API, so I guess this will require a lot of new API calls and structures? Some more open question to everyone else: Do we need Python bindings? Should libvirt's C API use/expose libparted structures directly? (And how would this affect the remote case?) Rich. -- Emerging Technologies, Red Hat http://et.redhat.com/~rjones/ 64 Baker Street, London, W1U 7DF Mobile: +44 7866 314 421 "[Negative numbers] darken the very whole doctrines of the equations and make dark of the things which are in their nature excessively obvious and simple" (Francis Maseres FRS, mathematician, 1759)

"Daniel P. Berrange" <berrange(a)redhat.com&gt; wrote: ... If the set of LVM operations required by libvirt is really so small, then I'd say it's worth investing in doing the right thing, if only to create the tiny API right away. With a real library API, libvirt stand a much better chance of properly diagnosing, and even detecting, partitioning failures. Of course, when everything works well, there's very little difference, but it's when things go wrong that users will thank us. Can you give me (an LVM newbie) an idea of what the few libvirt-required LVM operations are?

Richard W.M. Jones wrote: I should add, in a libvirt context it's probably going to be useful to also: * list available volume groups (vgscan) * list space available in each VG (vgdisplay name-of-vg) * show how VGs relate to PGs (pvscan) The LVM tools use quite a sophisticated internal API which isn't exported - but see for example tools/tools.h and include/*.h in the LVM2 source code. <libdevmapper.h> is also available, but much more lowlevel. Rich. -- Emerging Technologies, Red Hat http://et.redhat.com/~rjones/ 64 Baker Street, London, W1U 7DF Mobile: +44 7866 314 421 "[Negative numbers] darken the very whole doctrines of the equations and make dark of the things which are in their nature excessively obvious and simple" (Francis Maseres FRS, mathematician, 1759)

Richard W.M. Jones wrote: I talked to Alasdair Kergon about all of this on Tuesday, and there are a few clarifications. Firstly, vgscan & vgdisplay are apparently deprecated (who knew?) and there is a new command (vgs) which replaces both. The good news is that vgs is designed so that the output can be parsed: # /usr/sbin/vgs VG #PV #LV #SN Attr VSize VFree Home 1 1 0 wz--n- 298.09G 8.09G VolGroup00 1 2 0 wz--n- 152.56G 32.00M versus the old way: # /usr/sbin/vgdisplay Home --- Volume group --- VG Name Home System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 [etc] Similarly, pvs instead of pvscan/pvdisplay and lvs instead of lvscan/lvdisplay. Secondly there is an API of sorts for lvm2. I think Alasdair called it "libcmd", but maybe I got that wrong because Google doesn't seem to turn up anything. In any case, all it is is a wrapper around the command line tools, so it seems doubtful that this is going to be any better than just invoking the command line tools ourselves. Thirdly, Alasdair is quite keen to see a proper programming API for LVM2. However I don't think anyone has the time to do it, and he has the usual concerns for backwards binary compatibility and making sure the API mirrors the command line tools. LVM2 is still evolving at a fair rate. There are few other candidates to use the API at the moment, but apparently Anaconda does some really nasty stuff directly on LVM2 structures so it should be changed to use an API instead. My current opinion is that we can use libparted as it is to both manage virt-on-partition scenarios and to find orphan PVs as candidates to add to a VG, and the command line tools like vgs to manage and allocate LVM2 partitions. Rich. -- Emerging Technologies, Red Hat http://et.redhat.com/~rjones/ 64 Baker Street, London, W1U 7DF Mobile: +44 7866 314 421 "[Negative numbers] darken the very whole doctrines of the equations and make dark of the things which are in their nature excessively obvious and simple" (Francis Maseres FRS, mathematician, 1759)

"Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote: lvm2cmd is what you're looking for. It is only slightly higher-level than "system". Here's an example of that interface, from the lvm2 repo, doc/example_cmdlib.c: #include "lvm2cmd.h" /* All output gets passed to this function line-by-line */ void test_log_fn(int level, const char *file, int line, const char *format) { /* Extract and process output here rather than printing it */ if (level != 4) return; printf("%s\n", format); return; } int main(int argc, char **argv) { void *handle; int r; lvm2_log_fn(test_log_fn); handle = lvm2_init(); lvm2_log_level(handle, 1); r = lvm2_run(handle, "vgs --noheadings vg1"); /* More commands here */ lvm2_exit(handle); return r; }

Jim Meyering wrote: [...] Thanks for looking into this. From the virt-manager/libvirt p.o.v. it seems to me the important operations are: (1) Find attached drives. (2) Find partitions available & their sizes. (3) Allocate logical volumes. (4) Find out how much free space is available on a partition, and carve out a file. Correct me if I'm wrong (I usually am), but: Nothing can do (1) except doing a brute force scan over /dev and looking for likely block devices (this is what vgscan does). Parted can do (2), with several limitations including sector size. It can't do (3) at all, but then neither can anything else except forking the LVM command line tools. And (4) can be done by libvirtd using ordinary POSIX calls, so no external library support is needed, just some work to remote those operations (which is mostly done). Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903

"Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote: ... How about creating a regular partition, or resizing an existing (non-LVM) file system? Isn't doing #4 portably pretty tricky? There's still too much variation, because many of the details aren't covered by POSIX. At least for GNU df, it was -- it uses the mountlist module from gnulib: http://www.gnu.org/software/gnulib/MODULES.html the big pieces: http://cvs.sv.gnu.org/viewcvs/gnulib/lib/mountlist.c?root=gnulib&view... http://cvs.sv.gnu.org/viewcvs/gnulib/m4/ls-mntd-fs.m4?root=gnulib&vie... Of course, if your target is just Linux, then it is easier.

"Daniel P. Berrange" <berrange(a)redhat.com&gt; wrote: Lucky you :) Ok. Then this (also used by df) might help, if you ever need portability to e.g., older Solaris, *BSD, AIX, HP-UX, etc. http://cvs.sv.gnu.org/viewcvs/gnulib/lib/fsusage.c?root=gnulib&view=m... It provides a thin wrapper around statfs/statvfs, which helps handle some of the bogus values (sometimes negative or UINTMAX_MAX) that can arise in statvfs.f_* values. That last bit might even be useful on Linux.

Daniel P. Berrange wrote: I'm sure we can use it as guidance for possible problems though. I'm quite surprised there are portability problems with statfs. Isn't it a v7 call, didn't think there wouldn't be much that could go wrong :-) Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903

"Daniel P. Berrange" <berrange(a)redhat.com&gt; wrote: ... Ugh. That, again. Would you be seriously interested if it were LGPL? Since I own most of the pieces (the gnulib "fsusage" module does depend on several others) this might be enough to make me change it.

"Daniel P. Berrange" <berrange(a)redhat.com&gt; wrote: I've just switched the relevant gnulib modules to LGPL: http://thread.gmane.org/gmane.comp.lib.gnulib.bugs/10583 Let me know when/if you're interested in using it, and I'll be happy to lend a hand.

On Tue, Mar 20, 2007 at 12:06:39PM +0000, Mark McLoughlin wrote: I like the idea of storage pools too, but not the impl described in that thread :-) Creating a loop back mounted sparse file & running LVM on it is utterly disasterous for both performanceand & data integrity. It will also not be portable to any non-Linux systems which don't have LVM. It is really unneccessary too, since most host machines already have plenty of other ways for us to deal with storage - and importantly these are consistent with current manual approaches to managing storage so we have good compatability with non-libvirt managed storage. - There are a couple of different types of storage pool - An LVM volume group - Block devices - A directory on a filesystem - Each storage pool can have zero or more storage volumes allocated - LVM volume group has multiple logical volumes - Block device has multiple partitions - A directory has multiple files (maybe sparse) - Each storage pool has some measure of free space - LVM volume group has unallocated physical extents - Block device has unpartitioned sectors - A directory has free space from underlying filesystem - Every host has at least one storage pool with free space - ie a directory on a filesystem. Some hosts may also have free LVM space, or unpartitioned block devices but we can't assume their presence in general. This lets us manage all existing VMs which are either device (LVM/block) or file based (/var/lib/xen/images) with the new APIs, so gives a good back compatability story. The performance & reliability are good, since we're avoiding extra layers of loopback. There are only a handful of operations we need to track to get an initially useful API: - Enumerate storage pools - Enumerate volumes within a pool - Extract metadata about pools (free space, UUID?) - Extract metadata about volumes (logical size, physical allocation, UUID) - Create volume. - Delete volume That's pretty much it. I'd be inclined to implement regular file based pool in terms of /var/lib/xen/images (or /var/lib/libvirt/images?) as a first target. Its by far the easiest since it merely requires use of POSIX apis, and is also completely cross-platform portable (which LVM isn't). Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|

On Tue, Mar 20, 2007 at 02:58:16PM +0000, David Edmondson wrote: That all makes sense - the ZFS storage pools sounds like they provide equivalent volume management capabilities to what you'd get in LVM. Yes, sounds just fine. On this subject, does ZFS come with any library API for doing all the volume pool management tasks, or is it all just a set of command line tools as we'd get with LVM ? Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|

On Tue, 2007-03-20 at 14:32 +0000, Daniel P. Berrange wrote: /me too. But right now, we are not just lacking storage pools, we are also lacking reasonable local means to deal with storage. And robust support for handling storage locally is needed as the foundation for doing things like pools and non-local management. In addition, for pools we'd want to abstract away some of the differences between the various storage 'backends', i.e. you allocate a 10GB block device from pool 'foo' without having to worry whether that is backed by a container file or by an LV. That should probably be a layer that sits on top of the local API. I don't like putting block devices at this level; after all an LVM LV is a block device, too. And using partitions of a block device in analogy to LV's in a VG adds additional constraints, e.g. because the number of partitions is restricted. If a physical block device should become part of a pool, it should just be added to a volume group. And then there's stuff like SAN's, which is another way to carve physical storage into block devices, though I ahve no idea how we would dela with that at this level. That would be a really good first step for storage pools. David

On Thu, 2007-03-15 at 14:45 +0100, Jim Meyering wrote: There are several existing sets of python bindings for parted -- it seems to be relatively popular to just write your own rather than use an existing one ;-) One of the longest existing ones is pyparted which was originally written by Matt Wilson and is used extensively within anaconda and continues to be maintained by dcantrell. Jeremy

Hey, I know I said this already ... but it's probably worth fleshing out the idea a bit "on paper" to see how it works out. The idea is that perhaps we should support the concept of a "Virtual Storage Pool" in the same way we now support Virtual Networks. A virtual storage pool would basically be an area of storage on a physical machine from which virtual disks can be allocated for guests. It would be backed by e.g. an LVM Volume Group. On each host machine, there would be a default storage pool consisting of a large (e.g. 10G) sparse file loopback mounted and containing an LVM VG. Users can add other PVs to that VG in order to allocate more storage to the pool. So, what would the XML format be like for the simple case of a loopback sparse file? <storage_pool> <name>TestStorage</name> <uuid>1dd053b2-a068-4f6a-aaae-d7d88ecb504d</uuid> <pool type='lvm'> <physical_volume type='file'> <file size='20971520'>/var/lib/xen/test-storage.img</file> </physical_volume> </pool> </storage_pool> When you start create a pool, libvirtd would create the file, associate a loopback device with it, initialise it as a PV using pvcreate and create a volume group corresponding to the name of the pool using vgcreate. On subsequent boots, libvirtd would find that the file exists, associate a loopback device with the file, run vgscan and check that the VG exists[1]. (We could allow specifying an alternative VG name and having multiple physical volumes. Also note, that you'd only need to list physical volumes which actually need to be "activated" ... i.e. if the VG was just on /dev/hda3 or something, it wouldn't need to be listed) Of course, you then need to be able to carve out a chunk for a guest, so perhaps: int virStorageAllocateVolume(virStoragePtr storage, const char *name, unsigned long size); int virStorageDeallocateVolume(virStoragePtr storage, const char *name); Volumes could be allocated using the XML format too: <pool type='lvm'> <volume> <name>TestVolume</name> <size>4194304</name> </volume> </pool> And once allocated, you could create a guest with e.g. <disk type='volume'> <source volume='TestVolume' /> <target dev='hda' /> </disk> which would cause libvirtd to first lookup the device path for the volume and use that when starting the guest. Of course, we'd need to think about other types of physical volumes. The simple one is just a plain block device: <physical_volume type='device'> <file>/dev/hda3</file> </physical_volume> An iSCSI target: <physical_volume type='iscis'> <host>storage.devel.redhat.com</host> <port>3260</port> <target>iqn.1994-06.com.redhat.devel:markmc.test1</target> <lun>0</lun> </physical_volume> Or a file on an NFS mount: <physical_volume type='nfs'> <remote>storage.devel.redhat.com:/mnt/storage/test</remote> <file>test-storage.img</file> </physical_volume> Of course, all guests don't need to fit into this model they can continue using a file/device directly rather than a storage pool. Cheers, Mark. [1] - One thing to think about is that the VG contains the canonical list of physical volumes and allocated logical volumes ... so e.g. libvirt would be confused if you re-named a volume.