- Devel - libvirt List Archives

[Libvir] Concepts in storage management
by Daniel P. Berrange 22 Oct '07

22 Oct '07

So we've had many email threads on the subject of storage, but none have resulted in a satisfactory way forward to implementing any storage mgmt apis. Part of the problem I think is that we've not tried to understand all the various concepts / technologies which are available & how they relate to each other. This mail attempts to outline all the different technologies. There's a short list of API operations, but I don't want to particularly get into API details until we have a good undersatnding of the concepts. First and foremost I don't believe it is acceptable to say we're only going to allow one kind of storage. Storage is the key piece of infrastructure for any serious network, and we have to be able to adapt to deployment scenarios that present themselves. Second, there is clearly a huge number of storage technologies here and there's no way we'll implement support for all of them in one go. So we need to prioritize getting the conceptual model correct, to allow us to incrementally support new types of storage backend. Taxonomy of storage types ========================= | +- Block | | | +- Disk | | | | | +- Direct attached | | | | | | | +- IDE/ATA disk | | | +- SCSI disk | | | +- FibreChannel disk | | | +- USB disk/flash | | | +- FireWire disk/flash | | | | | +- Remote attached | | | | | +- iSCSI disk | | +- GNBD disk | | | +- Partition | | | +- Virtual | | | +- Direct attached | | | | | +- LVM | | +- ZFS | | | +- Remote attached | | | +- Cluster LVM | +- FileSystem | | | +- Directed attached | | | | | +- ext2/3/4 | | +- xfs | | +- ZFS | | | +- Remote attached | | | +- NFS | +- GFS | +- OCFS2 | +- Directory | +- File | +- Raw allocated +- Raw sparse +- QCow2 +- VMDK Storage attributes ================== - Local vs network (ext3 vs NFS, SCSI vs iSCSI) - Private vs shared (IDE vs FibreChannel) - Pool vs volume (LVM VG vs LV, Directory vs File, Disk vs Partition) - Container vs guest (OpenVZ vs Xen) - Attributes - Compressed - Encrypted - Auto-extend - Snapshots - RO - RW - Partition table - MBR - GPT - UUID - 16 hex digits - Unique string - SCSI WWID (world wide ID) - Local Path(s) (/dev/sda, /var/lib/xen/images/foo.img) - Server Hostname - Server Identifier (export path/target) - MAC security label (SELinux) - Redundancy - Mirrored - Striped - Multipath - Pool operation - RO - RW Nesting hierarchy ================= Many possibilities... - 1 x Host -> N x iSCSI target -> N x LUN -> N x Partition - N x Disk/Partition -> 1 x LVM VG -> B x LVM LV - 1 x Filesystem -> N x directory -> N x file - 1 x File -> 1 x Block (loopback) Application users ================= - virt-manager / virt-install - Enumerate available pools - Allocate volume from pool - Create guest with volume - virt-clone - Copy disks - Snapshot disks - virt-df - Filesystem usage - pygrub - Extract kernel/initrd from filesystem - virt-factory - Manage storage pools - Pre-migration sanity checks - virt-backup - Snapshot disks - virt-p2v - Snapshot disks Storage representation ====================== Two core concepts - Volume - a chunk of storage - assignable to a guest - assignable to a pool - optionally part of a pool - Pool - a chunk of storage - contains free space - allocate to provide volumes - compromised of volumes Recursive! n x Volume -> Pool -> n x Volume Nesting to many levels... Do we need an explicit Filesystem concept ? Operations ========== Limited set of operations to perform - List host volumes (physical attached devices) - List pools (logical volume groups, partitioned devs, filesystems) - List pool volumes (dev partitions, LVM logical volumes, files) - Define pool (eg create directory, or define iSCSI target) - Undefine pool (delete directory, undefine iSCSI config - Activate pool (mount NFS volume, login to iSCSI target) - Deactivate pool (unmount volume, logout of iSCSI) - Dump pool XML (get all the metadata) - Lookup by path - Lookup by UUID - Lookup by name - Create volume (create a file, allocate a LVM LV, etc) - Destroy volume (delete a file, deallocate a LVM LV) - Resize volume (grow or shrink volume) - Copy volume (copy data between volumes) - Snapshot volume (snapshot a volume) - Dump volume XML (get all the metadata) - Lookup by path - Lookup by UUID - Lookup by name http://www.redhat.com/archives/libvir-list/2007-February/msg00010.html http://www.redhat.com/archives/libvir-list/2007-September/msg00119.html Do we also need some explicit Filesystem APIs ? XML description =============== The horrible recursiveness & specific attributes are all in the XML description for different storage pool / volume types. This is where we define things like what physical volume are in a volume group, iSCSI server / target names, login details, etc, etc XXX fill in the hard stuff for metadata description here Implementation backends ======================= - FileSystem/Directory/File - POSIX APIs - LVM - LVM tools, or libLVM - Disk/partitions - sysfs / parted - iSCSI - sysfs / iscsi utils - ZFS - ZFS tools Implementation strategy ======================= Should prioritize implementation according to immediate application needs Initial goal to support remote guest creation on par with current capabilities: - Directory + allocateing raw sparse files - Enumerate existing disks, partitions & LVM volumes Further work: - Allocating LVM volumes - Defining LVM volume groups - Partitioning disks - Mounting networked filesystems - Accessing iSCSI volumes - Copying existing volumes - Snapshotting volumes - Cluster aware filesystems (GFS) - Various file formats (QCow, VMDK, etc) Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|

4 13

Re: [Libvir] Problem to compile virt-manager in debian etch
by Richard W.M. Jones 19 Oct '07

19 Oct '07

Marco Sinhoreli wrote: > The configure script was found, but the Makefile hasn't been created. Need to see the full messages. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903

4 8

[Libvir] [PATCH] fix negative value handling of virConfParseLong()
by Tatsuro Enokura 19 Oct '07

19 Oct '07

Hi, I found a bug about negative value at virConfParseLong() in conf.c. This patch fixes it. Signed-off-by: Tatsuro Enokura <fj7716hz(a)aa.jp.fujitsu.com> Thanks, Tatsuro Enokura. -------------------------------------------------------------------------- Index: libvirt/src/conf.c =================================================================== RCS file: /data/cvs/libvirt/src/conf.c,v retrieving revision 1.11 diff -u -p -r1.11 conf.c --- libvirt/src/conf.c 4 Jul 2007 09:16:23 -0000 1.11 +++ libvirt/src/conf.c 19 Oct 2007 08:37:36 -0000 @@ -345,6 +345,8 @@ virConfParseLong(virConfParserCtxtPtr ct l = l * 10 + (CUR - '0'); NEXT; } + if (neg) + l *= -1; *val = l; return(0); } --------------------------------------------------------------------------

3 2

[Libvir] Opening KVM guest in xterm rather than vnc
by David Mueller 19 Oct '07

19 Oct '07

I'm trying to setup my virtual machines to display in an xterm window rather than a VNC window. The guests and host are Fedora 7; the guest only needs to run in a console mode (no X11). I see a bit on http://libvirt.org/format.html about using a console tag instead of graphics in my XML, but I can't figure out how to connect to that. With our older Xen on Fedora Core 5 setup, we used the following command to create a guest and it worked great; I'm trying to approximate something similar with libvirt/KVM on Fedora 7: xterm -e "/usr/sbin/xm create -c guest.xen" A notable advantage of using an xterm over VNC is that we can resize the window; with VNC as far as I've been able to see, we've been stuck with the default 80x24 window. We also can't copy and paste text from the VNC window to other applications. Any suggestions would be appreciated. I am currently using libvirt-0.3.2-1.fc7; if this is something that we'll have to wait for a newer version that will be in Fedora 8, that's okay. - David

3 2

[Libvir] memory error in another test
by Jim Meyering 19 Oct '07

19 Oct '07

Running tests under valgrind exposed this: (line numbers are slightly off, due to local changes) ==27876== ERROR SUMMARY: 32 errors from 2 contexts (suppressed: 5 from 2) ==27876== ==27876== 16 errors in context 1 of 2: ==27876== Conditional jump or move depends on uninitialised value(s) ==27876== at 0x41BA18: qemudBuildCommandLine (qemu_conf.c:1871) ==27876== by 0x406764: testCompareXMLToArgvFiles (qemuxml2argvtest.c:46) ==27876== by 0x4069DA: testCompareXMLToArgvHelper (qemuxml2argvtest.c:97) ==27876== by 0x406D60: virtTestRun (testutils.c:69) ==27876== by 0x406A46: main (qemuxml2argvtest.c:114) ==27876== ==27876== 16 errors in context 2 of 2: ==27876== Conditional jump or move depends on uninitialised value(s) ==27876== at 0x41AA45: qemudBuildCommandLine (qemu_conf.c:1606) ==27876== by 0x406764: testCompareXMLToArgvFiles (qemuxml2argvtest.c:46) ==27876== by 0x4069DA: testCompareXMLToArgvHelper (qemuxml2argvtest.c:97) ==27876== by 0x406D60: virtTestRun (testutils.c:69) ==27876== by 0x406A46: main (qemuxml2argvtest.c:114) Both of those are due to uses of uninitialized vm->migrateFrom[0]. Here's the fix: Avoid memory error in a test program. * tests/qemuxml2argvtest.c (testCompareXMLToArgvFiles): Initialize vm.migrateFrom[0], to avoid "read-uninitialized" error from within qemudBuildCommandLine. Index: tests/qemuxml2argvtest.c =================================================================== RCS file: /data/cvs/libvirt/tests/qemuxml2argvtest.c,v retrieving revision 1.5 diff -u -p -r1.5 qemuxml2argvtest.c --- tests/qemuxml2argvtest.c 21 Sep 2007 21:20:32 -0000 1.5 +++ tests/qemuxml2argvtest.c 17 Oct 2007 13:42:15 -0000 @@ -40,6 +40,7 @@ static int testCompareXMLToArgvFiles(con vm.qemuVersion = 0 * 1000 * 100 + (8 * 1000) + 1; vm.qemuCmdFlags = QEMUD_CMD_FLAG_VNC_COLON | QEMUD_CMD_FLAG_NO_REBOOT; + vm.migrateFrom[0] = '\0'; vmdef->vncActivePort = vmdef->vncPort;

2 2

[Libvir] avoid reading uninitialized memory in test program
by Jim Meyering 19 Oct '07

19 Oct '07

valgrind complained about the printf in conftest.c. Here's the fix: * src/conf.c (__virConfWriteMem): Fix typos in comment. * tests/conftest.c (main): Use fwrite, not printf, since the result buffer is not NUL-terminated. =================================================================== RCS file: /data/cvs/libvirt/src/conf.c,v retrieving revision 1.11 diff -u -p -r1.11 conf.c --- src/conf.c 4 Jul 2007 09:16:23 -0000 1.11 +++ src/conf.c 17 Oct 2007 11:31:54 -0000 @@ -1,7 +1,7 @@ /** * conf.c: parser for a subset of the Python encoded Xen configuration files * - * Copyright (C) 2006 Red Hat, Inc. + * Copyright (C) 2006, 2007 Red Hat, Inc. * * See COPYING.LIB for the License of this software * @@ -916,7 +916,7 @@ error: /** * __virConfWriteMem: * @memory: pointer to the memory to store the config file - * @len: pointer to the lenght in byte of the store, on output the size + * @len: pointer to the length in bytes of the store, on output the size * @conf: the conf * * Writes a configuration file back to a memory area. @len is an IN/OUT =================================================================== RCS file: /data/cvs/libvirt/tests/conftest.c,v retrieving revision 1.1 diff -u -p -r1.1 conftest.c --- tests/conftest.c 29 Aug 2006 22:27:07 -0000 1.1 +++ tests/conftest.c 17 Oct 2007 11:31:54 -0000 @@ -24,7 +24,10 @@ int main(int argc, char **argv) { fprintf(stderr, "Failed to serialize %s back\n", argv[1]); exit(3); } - printf("%s", buffer); virConfFree(conf); + if (fwrite(buffer, len, 1, stdout) != len) { + fprintf(stderr, "Write failed\n"); + exit(1); + } exit(0); }

2 2

[Libvir] PATCH: RFC supporting SASL authentication
by Daniel P. Berrange 18 Oct '07

18 Oct '07

History: we currently provide two TCP sockets, one clear text, no auth, the other TLS with x509 client certificate verification for auth. We also provide a UNIX domain socket relying on file perms to restrict access. I have previously provided a policykit patch for the latter allowing admin defined auth policy for UNIX socket access. The PolicyKit patch was flawed in that it did not provide a way for a client app to determine whether the UNIX socket needed PolicyKit auth ahead of time or not. This required apps to make assumptions prior to connecting which is not really viable. This mail is working towards a more flexible authentication solution, and goes straight for the big picture by integrating SASL authentication. This gives us integration with Kerberos (GSSAPI) and PAM and whatever the hell else SAS supports[1]. The critical decision in all this is the wire protocol & how to adapt it to slot in authentication in a way that is compatible with existing clients. As it stands the decision to use clear data vs TLS on the wire is based on the socket the client connects to - we do a TLS handshake at the very start. We need to be able to support SASL on all sockets, TCP, TLS & UNIX since they can all benefit from stuff like Kerberos. So I decided we need to integrate at the RPC layer for this to work. The general protocol picture ---------------------------- It helps to look at qemud/remote_protocol.x when reading this next ... Every API basically has an RPC call & reply pair. The reply messages have a status field, either 'REMOTE_OK' or 'REMOTE_ERROR'. In the former case the normal API return values follow on the wire. In the latter case a virErrroPtr object is serialized on the wire. For this patch I decided to add a 3rd code, REMOTE_AUTH. If an application tries to make an RPC call on a socket requiring authentication, and has not yet authenticated the server will return REMOTE_AUTH code. It then also returns an int (remote_auth_type) specifying the authentication method to use. I have defined two: enum remote_auth_type { REMOTE_AUTH_NONE = 0, REMOTE_AUTH_SASL = 1 }; With plans to add REMOTE_AUTH_POLKIT in a future patch. A legacy client getting back REMOTE_AUTH code will just quit the connection attempt since they don't support authentication. If the admin so desires they can still provide the TLS socket in a non-authenticated mode and only turn on SASL for the TCP socket. So the decision about whether to enable legacy clients is admin controlled. This is the best we can do. A new client getting back REMOTE_AUTH code will then read the remote_auth_type off the wire. If the requested type is one that the client supports then it can begin the authentication process, otherwise we virRaiseError and stop connecting. The SASL specific picture ------------------------- For the SASL auth the process involves a multi-phase handshake looking something like: client server 1. -> ask for mechanism list -> new ctx 2. new ctx <- list of mechanisms <- 3. start auth -> initial auth data -> start auth 4. step auth <- reply auth data <- 5. -> step auth data -> step auth goto 4. <- reply auth data <- Authentication can complete at step 4 in this process, or steps 4 & 5 can repeat an arbitrary number of times. So, to implement this if it neccessary to define 3 new RPC calls internal to the remote driver/daemon (aka not exposed to public API like the rest of the RPC calls). These are: REMOTE_PROC_AUTH_SASL_INIT = 66, REMOTE_PROC_AUTH_SASL_START = 67, REMOTE_PROC_AUTH_SASL_STEP = 68 These are basically just punting back & forth the data going in & out of the appropriate SASL apis. sasl_{server,client}_{new,start,step} See the man pages for more info. So on the server end, if a socket is configured to require SASL auth, the server will reject all RPC calls *except* for those three above with the REMOTE_AUTH code. Once SASL auth has completed, it will allow all the normal RPC calls. The effect is basically that the client is not able to call virConnectOpen & friends until auth has completed. On the client end, the fun is in the 'call' method of remote_internal.c. This has been split in two. The original method is now 'onecall', and there is a thin wrapper named 'call'. 'call' simply invokves onecall, and if it gets back a REMOTE_AUTH code, will do the SASL handshake & then re-run the original call. So again the effect is basically that the first virConnectOpen will cause the auth handshake to be performed. The SASL implementation details ------------------------------- This is the bare minimum SASL integration. I have not attempted to hook up any callbacks for gathering credentials. This basically means that the only SASL mechanism which works is Kerberos GSSAPI - its credentials are fetched out-of-band & so don't require callbacks. We do need to consider callbacks later so we can do username/password auth, and all the various other methods SASL has. As well as authentication, some SASL mechanisms provide a way to negotiate a data encryption layer for the subsequent session. GSSAPI is the only commonly used mechanism which supports this. I have not implemented this yet though. What we would do though is to enable this capability on the the plain TCP socket only. This would make the TCP socket truely secure, and avoid any extra overhead on the TLS socket or UNIX domain socket. I have set the wire packet size for the SASL negotiation to 8192 bytes at a time. This has been sufficient so far, but I need to validate this before we commit, because this will be wire ABI sensitive. Or I could change the XDR spec to be variable length. Anyway needs re-visiting This only deals with authentication. I have not attempted any authoriztion controls. So anyone who has a valid kerberos principle can connect to the server. We clearly need a local group list as we do for the x509 client certificates. Ultimately we could try LDAP lookups & other intersting suggestions. The SASL stuff is detected in configure & enabled/disabled as appropriate. I need to add extra config file params though to let the sysadmin control what socket uses what auth mechanism. The setup / usage details ------------------------- As I said, I only used GSSAPI so far. To use this all your clients need to be able to kinit & get a ticket. For testing I have setup my own personal Kerberos server using Fedora 7 & FreeIPA (http://freeipa.org/) Each libvirt server needs to be issued with a Kerberos service principle. I am using the word 'libvirt' as the service name. The service principle must match the FQDN of the server host. So on your Kerberos server you can issue a service principle with kadmin.local > addprinc libvirt/cherry.virt.boston.redhat.com(a)VIRT.BOSTON.REDHAT.COM > ktadd -k /tmp/cherry-libvirt.tab libvirt/cherry.virt.boston.redhat.com(a)VIRT.BOSTON.REDHAT.COM > quit Then copy the /tmp/cherry-libvirt.tab file to /etc/libvirt/krb5.tab on the server in question. (Change the hostnames & REALM as needed of course). If you want to change the GSSAPI mechanism, the settings are in the file /etc/sasl2/libvirt.conf. Though again only GSSAPI works so far. If you edit the /etc/libvirt/libvirtd.conf file you can enable listen_tcp=1 and run run libvirtd --listen. >From the client machine it should now be possible to do $ kinit berrange(a)VIRT.BOSTON.REDHAT.COM $ virsh --connect xen+tcp://cherry.virt.boston.redhat.com/ You probably screwed something up though along the way because Kerberos is nasty like that. If you use --verbose with libvirtd it'll show details of any errors during authentication. On the client end you can add in a param on the connect URI of ?debug=stderr and it'll print some info about the auth process to stderr. Check that it shows 'GSSAPI' as a valid mechanism. If it doesn't, then your server is not configured as it should be - check keytab file - check you have cyrus-sasl-gssapi RPM. If that's ok, then check 'klist' on the client - if the first phase of chatter between the client & the KDC worked you should see the principle of the server cached on the client. If you don't, then check the KDC logs in /var/log/krb5kdc.log Kerberos/GSSAPI error reporting sucks really bad. That said, I'm sure I'm missing something, because the errors I get out of SASL are even worse than normal. BTW, you can see various 'XXX' in my patch. Basically all of them need to be addressed before I'd consider this patch suitable to merge. configure.in | 39 +++ include/libvirt/virterror.h | 1 libvirt.spec.in | 3 qemud/Makefile.am | 21 +- qemud/internal.h | 8 qemud/libvirtd.init.in | 3 qemud/libvirtd.sysconf | 3 qemud/qemud.c | 29 ++ qemud/remote.c | 370 ++++++++++++++++++++++++++++++++++-- qemud/remote_dispatch_localvars.h | 5 qemud/remote_dispatch_proc_switch.h | 24 ++ qemud/remote_dispatch_prototypes.h | 3 qemud/remote_protocol.c | 164 +++++++++++++++ qemud/remote_protocol.h | 59 +++++ qemud/remote_protocol.x | 53 ++++- src/Makefile.am | 3 src/remote_internal.c | 307 +++++++++++++++++++++++++++++ src/virsh.c | 4 src/virterror.c | 6 tests/Makefile.am | 2 20 files changed, 1073 insertions(+), 34 deletions(-) Regards, Dan. [1] Not entirely true. Some auth methods require credentials to be fetched from the user, so we can only support methods for which we have callbacks. This patch ignores the callback question, so only support GSSAPI where the credentials are fetched out-of-band. This can be addressed later. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|

2 3

Re: [Libvir] Problem to compile virt-manager in debian etch
by Richard W.M. Jones 18 Oct '07

18 Oct '07

Marco Sinhoreli wrote: > The autogen.sh output: > > ./autogen.sh > libtoolize: `config.guess' exists: use `--force' to overwrite > libtoolize: `config.sub' exists: use `--force' to overwrite > libtoolize: `ltmain.sh' exists: use `--force' to overwrite > hecking for a BSD-compatible install... /usr/bin/install -c > checking whether build environment is sane... yes > checking for a thread-safe mkdir -p... /bin/mkdir -p > checking for gawk... no > checking for mawk... mawk > [...] > > The remaining seems okay. So looks like it's working then? Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903

2 1

[Libvir] storage API and ZFS
by John Levon 18 Oct '07

18 Oct '07

Accidentally deleted your reply Richard, sorry. ZFS 'pools' are aggregates composed of some number of real devices. LVM confuses me so I don't know what they map on to, but the basic idea is that you can carve filesystems out of such pools at will. You can also carve out 'volumes' - these appear as a disk device in /dev/zvol/dsk/poolname/volumename It's the latter that you use in Xen to hold domU root filesystems (or you can use a normal file on whatever file system, or a real device, as desired). What might be tricky is anything that claims to return "space left", depending exactly on what you want it to mean. But (I think) certain LVM setups have similar issues. regards john

2 1

[Libvir] Is it possible to add an interface to intercept the migration process?
by Meng Ran 18 Oct '07

18 Oct '07

Hi All, I'm thinking about if it possible to add an interface to let user specified action to happen on destination during the migration, this should be useful for migrating things that have different paths on source and destination. Allowing specified action(callback) to happen at a time point, e.g., after the config file is transfered and before starting the vm, can bring in a lot of flexibilities considering the migration of devices which are shared but have different paths/views. Giving the dest a chance to recognize these devices and update the config, and then start the vm probably is helpful there. Comment? Thanks, Jack

5 6