Re: [libvirt] [PATCH 0/1] Multipath pool support

Daniel P. Berrange wrote: > On Thu, Jul 23, 2009 at 02:53:48PM -0400, Dave Allan wrote: >> Daniel P. Berrange wrote: >>>> It doesn't currently allow configuration of multipathing, so for >>>> now setting the multipath configuration will have to continue to be >>>> done as part of the host system build. >>>> >>>> Example XML to create the pool is: >>>> >>>> <pool type="mpath"> <name>mpath</name> <target> >>>> <path>/dev/mapper</path> </target> </pool> >>> So this is in essence a 'singleton' pool, since there's only really >>> one of them per host. There is also no quanity of storage associated >>> with a mpath pool - it is simply dealing with volumes from other >>> pools. This falls into the same conceptual bucket as things like >>> DM-RAID, MD-RAID and even loopback device management. >> It is a singleton pool, in that there is only one dm instance per host. >> With regard to capacity, the dm devices have capacity, and their >> constituent devices could be members of other pools. Can you elaborate >> on what you see as the implications of those points? > > The storage pool vs storage volume concept was modelled around the idea > that you have some storage source, and it is sub-divided into a number > of volumes > > With multipath pool you have no storage source - the source is the > SCSI/iSCSI pool which actually provides the underlying block provides > which are the LUN paths. So by having a explicit storage pool for > multipath, there's an implicit dependancy between 2 pools. If you > refresh a SCSI pool, you must then refresh a multipath pool too. > Or if you add a SCSI/iSCSI pool you must also refresh the multipath > pool. There's also the issue of tracking the assoication between > multipath volumes and the pools to ensure you don't remove a pool > that's providing a multipath volume thats still in use. The problem of hierarchical relationships among pools can exist with the other pools as well, since one could create a logical pool on top of a block device that's part of an iSCSI or other pool. It's also possible that a hierarchical pool relationship might not exist with the multipath pool if a user didn't create pools for HBAs. >>> The question I've never been able to satisfactorily answer myself is >>> whether these things(mpath,raid,loopback) should be living in the >>> storage pool APIs, or in the host device APIs. >>> >>> I also wonder people determine the assoication between the volumes in >>> the mpath pool, and the volumes for each corresponding path. eg, how >>> do they determine that /dev/mapper/dm-4 multipath device is >>> associated with devices from the SCSI storage pool 'xyz'. The storage >>> volume APIs & XML format don't really have a way to express this >>> relationship. >> It's not difficult to query to find out what devices are parents of a >> given device, but what is the use case for finding out the pools of the >> parent devices? > > Say you have 3 SCSI NPIV pools configured, and a multipath pool. > You want to remove one of the SCSI pools, and know that the > multipath devices X, Y & Z are in use. You need to determine which > of the SCSI pools contains the underlying block devices for these > multipath devices before you can safely remove that SCSI pool. Ok, that makes sense, but this problem exists with any hierarchical pool so users are already dealing with it. >>> The host device APIs have a much more limited set of operations >>> (list, create, delete) but this may well be all that's needed for >>> things like raid/mpath/loopback devices, and with its XML format >>> being capability based we could add a multipath capability under >>> which we list the constituent paths of each device. >> If we decide to implement creation and destruction of multipath devices, >> I would think the node device APIs would be the place to do it. > > If we intend to do creation/deletion of multipath devices in the > node device APIs, then we esentially get listing of multipath > devices in the node device APIs for free. So do we need a dedicated > storage pool for multipath too ? Isn't the general idea that storage pools are how people should be managing storage? We shouldn't make people use a separate API to enumerate one type of storage. > I have a feeling that the DeviceKit impl of the node devive APIs (which > is currently disabled by default, may already be reporting on all > device mapper block devices - the HAL impl does not. That may be--there's a fairly wide gap between the two sets of functionality. >>> Now, if my understanding is correct, then if multipath is active it >>> should automatically create multipath devices for each unique LUN on >>> a storage array. DM does SCSI queries to determine which block >>> devices are paths to the same underlying LUN. >> That's basically correct, and the administrator can configure which >> devices have multipath devices created. >> >>> Taking a simple iSCSI storage pool >>> >>> <pool type='iscsi'> <name>virtimages</name> <source> <host >>> name="iscsi.example.com"/> <device path="demo-target"/> </source> >>> <target> <path>/dev/disk/by-path</path> </target> </pool> >>> >>> this example would show you each individual block device, generating >>> paths under /dev/disk/by-path. >>> >>> Now, we decide we want to make use of multipath for this particular >>> pool. We should be able to just change the target path, to point to >>> /dev/mpath, >>> >>> <pool type='iscsi'> <name>virtimages</name> <source> <host >>> name="iscsi.example.com"/> <device path="demo-target"/> </source> >>> <target> <path>/dev/mpath</path> </target> </pool> >>> >>> and have it give us back the unique multipath enabled LUNs, instead >>> of each individual block device. >> The problem with this approach is that dm devices are not SCSI devices, >> so putting them in a SCSI pool seems wrong. iSCSI pools have always >> contained volumes which are iSCSI block devices, directory pools have >> always had volumes which are files. We shouldn't break that assumption >> unless we have a good reason. It's not impossible to do what you >> describe, but I don't understand why it's a benefit. > > What is a SCSI device though ? Under Linux these days everything appears > to be a SCIS device whether it is SCSI or not, eg PATA, SATA, USB. So > there can be no assumption that a SCSI HBA pool gives you SCSI devices. > If an application using a pool expects volumes to have particular > SCSI capabilities (peristent reservations for example), then the only > way is for it to query the device, or try the capability it wants and > handle failure. The best libvirt can guarentee is that a SCSI, disk, > iSCSI & logical pools will give back block devices, while fs / netfs > pools will give back plain files. > The one downside I realize with my suggestion here, is that a single > multipath device may have many paths, and each path may go via a > separate HBA, which would mean separate SCSI pool. So in fact I think > we shouldn't expose multipath in normal SCSI pools after all :-) Agreed, let's keep the existing pools the way they are. > I'm still inclined to think we can do the 'list' operation in node device > APis though Again, I think using the node device APIs as the only support for multipath devices is contrary to how we're leading people to believe storage should be managed with libvirt. >>>> The target element is ignored, as it is by the disk pool, but the >>>> config code rejects the XML if it does not exist. That behavior >>>> should obviously be cleaned up, but I think that should be done in >>>> a separate patch, as it's really a bug in the config code, not >>>> related to the addition of the new pool type. >>> The target element is not ignored by the disk pool. This is used to >>> form the stable device paths via virStorageBackendStablePath() for >>> all block device based pools. >> Hmm--on my system the path I specify shows up in the pool XML, but is >> unused as far as I can tell. I can hand it something totally bogus and >> it doesn't complain. I think your next point is very good, though, so >> I'll make the target element meaningful in the multipath case and we can >> investigate the disk behavior separately. > > Normally a disk pool will give you back volumes whose path name > is /dev/sdXX. If you give the pool a target path if /dev/disk/by-uuid > then the volumes will get paths like > /dev/disk/by-uuid/b0509f5a-2824-4090-9da2-d0f0ff4ace0e > Since it is possible that some volumes may not have stable paths > though, we fall back to /dev/sdXXX if one can't be formed. > > We should probably explicitly reject bogus target paths which don't > even exist on disk though. Only allow targets under /dev, where the > given target exists That sounds good. Dave