[Libvir] ISCSI howto, example, etc?
The documentation on the new storage feature is either a little short, or I totally missed it. Anyway, I am looking for some help on how to use it. If anyone could maybe give me an example xml file that would be incredibly helpful, because based on the format description I can't exactly figure out what information I am supposed to put where (for ISCSI). Thanks! -- $ ./Mark
On Tue, May 06, 2008 at 12:50:25AM -0600, Mark Dehus wrote:
The documentation on the new storage feature is either a little short, or I totally missed it. Anyway, I am looking for some help on how to use it. If anyone could maybe give me an example xml file that would be incredibly helpful, because based on the format description I can't exactly figure out what information I am supposed to put where (for ISCSI).
The main docs we have are http://libvirt.org/formatstorage.html http://libvirt.org/storage.html They are basically more reference guides that howto/tutorials. Also look at the 'virsh help' output for 'vol-XXX' and 'pool-XXX' commands. As a simple example... - Create an XML definition for the iSCSI target you want to access, # cat > virt.xml <<EOF <pool type="iscsi"> <name>virtimages</name> <source> <host name="iscsi.example.com"/> <device path="demo-target"/> </source> <target> <path>/dev/disk/by-path</path> </target> </pool> EOF - Define the pool # virsh pool-define virt.xml - Activate the pool (this step logs into the server) # virsh pool-start virtimages - List the LUNs available # virsh vol-list virtimages - Query info about a LUN # virsh vol-info --pool virtimages lun-3 # virsh vol-dumpxml --pool virtimages lun-3 There's many more pool-XXX and vol-XX commands available Regards, Dan. -- |: Red Hat, Engineering, Boston -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Tue, 6 May 2008, Daniel P. Berrange wrote:
As a simple example...
- Create an XML definition for the iSCSI target you want to access,
I have created: <pool type="iscsi"> <name>netapp</name> <source> <host name="172.16.103.200"/> <device path="iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f"/> </source> <target> <path>/dev/disk/by-path</path> </target> </pool>
- Define the pool
# virsh pool-define virt.xml
xen01 netapp # virsh pool-define virt.xml Pool netapp defined from virt.xml
- Activate the pool (this step logs into the server)
# virsh pool-start virtimages
xen01 netapp # virsh pool-start netapp Pool netapp started
- List the LUNs available
# virsh vol-list virtimages
xen01 netapp # virsh vol-list netapp Name Path ----------------------------------------- << pretty empty >> But when I look in /dev/disk/by-path: lrwxrwxrwx 1 root root 9 May 14 12:35 ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-0 -> ../../sdc lrwxrwxrwx 1 root root 9 May 14 12:35 ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-1 -> ../../sdd lrwxrwxrwx 1 root root 9 May 14 12:35 ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-2 -> ../../sde Something is present, thus the login was succesfull. How to get this to work with the libvirt storage pool? I have tried to change the target path. But it doesn't give me any output. I'll now look in the C code to see what you are giving as command for this list. Stefan
On Wed, 14 May 2008, Stefan de Konink wrote:
Something is present, thus the login was succesfull. How to get this to work with the libvirt storage pool?
I have tried to change the target path. But it doesn't give me any output. I'll now look in the C code to see what you are giving as command for this list.
static int virStorageBackendISCSIFindLUNs(virConnectPtr conn, virStoragePoolObjPtr pool, const char *session) { /* * # iscsiadm --mode session -r $session -P 3 * * scsi1 Channel 00 Id 0 Lun: 0 * scsi1 Channel 00 Id 0 Lun: 1 * Attached scsi disk sdc State: running * scsi1 Channel 00 Id 0 Lun: 2 * Attached scsi disk sdd State: running * scsi1 Channel 00 Id 0 Lun: 3 * Attached scsi disk sde State: running * scsi1 Channel 00 Id 0 Lun: 4 * Attached scsi disk sdf State: running * scsi1 Channel 00 Id 0 Lun: 5 * Attached scsi disk sdg State: running * * Need a regex to match the Channel:Id:Lun lines */ I consider my output interesting... it seems the -P option is ignored. (I'm running open-iscsi-2.0.868_rc1) xen01 src # iscsiadm --mode session -r 10 -P 3 node.name = iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f node.tpgt = 2003 node.startup = manual iface.hwaddress = default iface.iscsi_ifacename = default iface.net_ifacename = default iface.transport_name = tcp node.discovery_address = 172.16.103.200 node.discovery_port = 3260 node.discovery_type = send_targets node.session.initial_cmdsn = 0 node.session.initial_login_retry_max = 4 node.session.cmds_max = 128 node.session.queue_depth = 32 node.session.auth.authmethod = None node.session.auth.username = <empty> node.session.auth.password = <empty> node.session.auth.username_in = <empty> node.session.auth.password_in = <empty> node.session.timeo.replacement_timeout = 120 node.session.err_timeo.abort_timeout = 15 node.session.err_timeo.lu_reset_timeout = 20 node.session.err_timeo.host_reset_timeout = 60 node.session.iscsi.FastAbort = Yes node.session.iscsi.InitialR2T = No node.session.iscsi.ImmediateData = Yes node.session.iscsi.FirstBurstLength = 262144 node.session.iscsi.MaxBurstLength = 16776192 node.session.iscsi.DefaultTime2Retain = 0 node.session.iscsi.DefaultTime2Wait = 2 node.session.iscsi.MaxConnections = 1 node.session.iscsi.MaxOutstandingR2T = 1 node.session.iscsi.ERL = 0 node.conn[0].address = 172.16.103.200 node.conn[0].port = 3260 node.conn[0].startup = manual node.conn[0].tcp.window_size = 524288 node.conn[0].tcp.type_of_service = 0 node.conn[0].timeo.logout_timeout = 15 node.conn[0].timeo.login_timeout = 15 node.conn[0].timeo.auth_timeout = 45 node.conn[0].timeo.noop_out_interval = 5 node.conn[0].timeo.noop_out_timeout = 5 node.conn[0].iscsi.MaxRecvDataSegmentLength = 131072 node.conn[0].iscsi.HeaderDigest = None,CRC32C node.conn[0].iscsi.DataDigest = None node.conn[0].iscsi.IFMarker = No node.conn[0].iscsi.OFMarker = No
On Wed, 14 May 2008, Stefan de Konink wrote:
I consider my output interesting... it seems the -P option is ignored. (I'm running open-iscsi-2.0.868_rc1)
After upgrading to open-iscsi-2.0.869.2 i have a working output with -P 3... but... xen01 skinkie # virsh pool-start netapp libvir: Storage error : Failed to find the sysfs path for 4:0:0:2: No such file or directory error: Failed to start pool netapp That doesn't make me happy. snprintf(sysfs_path, PATH_MAX, "/sys/class/iscsi_session/session%s/device/" "target%d:%d:%d/%d:%d:%d:%d/block", session, target, channel, id, target, channel, id, lun); xen01 5:0:0:1 # pwd /sys/class/iscsi_session/session5/device/target5:0:0/5:0:0:1 xen01 5:0:0:1 # ls -l total 0 lrwxrwxrwx 1 root root 0 May 14 13:14 block:sdd -> ../../../../../../block/sdd I guess this is not what makes people happy. (:realdisk) Is this something we can solve or should we call the open-iscsi guys? Stefan
On Wed, 14 May 2008, Stefan de Konink wrote:
That doesn't make me happy.
I have currently solved it and made it work with the new version, I'll send in a patch. But there are more things that don't make me happy. 1) lun-0 is ignored 2) Rescanning session [sid: 8, target: iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f, portal: 172.16.103.200,3260] ********** sde ********** sdd Link '/dev/disk/by-path/ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-2' does not point to '/dev/sdd', ignoring xen01 libvirt # virsh vol-list netapp Name Path ----------------------------------------- lun-1 /dev/disk/by-path/ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-1 lun-2 /dev/disk/by-path/ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-2 So something is still not how I want it to be. Stefan
Stefan de Konink wrote:
snprintf(sysfs_path, PATH_MAX, "/sys/class/iscsi_session/session%s/device/" "target%d:%d:%d/%d:%d:%d:%d/block", session, target, channel, id, target, channel, id, lun);
xen01 5:0:0:1 # pwd /sys/class/iscsi_session/session5/device/target5:0:0/5:0:0:1 xen01 5:0:0:1 # ls -l total 0 lrwxrwxrwx 1 root root 0 May 14 13:14 block:sdd -> ../../../../../../block/sdd
Ug. Yeah, that will do it. What kernel version are you running that your sysfs looks like that? It seems we have been shoved into a corner; we can't rely on the output of the iscsiadm tools, since we have already seen those change between versions. And apparently we can't rely on sysfs paths to stay stable either. Either we need some iscsiadm command with a known, stable output that we can call, or we need some sort of iSCSI API that we can call to get this information. Either way, I think we need help from the iSCSI people, because the current position seem to be untenable. Chris Lalancette
On Wed, 14 May 2008, Chris Lalancette wrote:
Ug. Yeah, that will do it. What kernel version are you running that your sysfs looks like that?
I already send a message to open-iscsi group. Fix is trivial and even more simple. I'm running 2.6.21-xen (gentoo) with open-iscsi-2.0.869.2. Fix: snprintf(sysfs_path, PATH_MAX, "/sys/class/iscsi_session/session%s/device/" "target%d:%d:%d/%d:%d:%d:%d", session, target, channel, id, target, channel, id, lun); (so remove /block) /* OK, not . or ..; let's see if it is a SCSI device */ if (len > 8 && block_dirent->d_name[0] == 'b' && block_dirent->d_name[1] == 'l' && block_dirent->d_name[2] == 'o' && block_dirent->d_name[3] == 'c' && block_dirent->d_name[4] == 'k' && block_dirent->d_name[5] == ':' && block_dirent->d_name[6] == 's' && block_dirent->d_name[7] == 'd') { /* looks like a scsi device, smells like scsi device; it must be a scsi device */ dev = (char *) calloc(sizeof(char), len - 5); strncpy(dev, &(block_dirent->d_name[6]), (len - 6)); I guess that can be come a strncmp. And for sake of memory management an if (dev != NULL) would be good too.
It seems we have been shoved into a corner; we can't rely on the output of the iscsiadm tools, since we have already seen those change between versions. And apparently we can't rely on sysfs paths to stay stable either. Either we need some iscsiadm command with a known, stable output that we can call, or we need some sort of iSCSI API that we can call to get this information. Either way, I think we need help from the iSCSI people, because the current position seem to be untenable.
I would love to see something like a iSCSI API. The parsing looks really cute, but as you say it is so unreliable. Next to this: /* the 0'th LUN isn't a real LUN, it's just a control LUN; skip it */ I think this is also a very strange idea of LUNs. Stefan
Stefan de Konink wrote:
Fix: snprintf(sysfs_path, PATH_MAX, "/sys/class/iscsi_session/session%s/device/" "target%d:%d:%d/%d:%d:%d:%d", session, target, channel, id, target, channel, id, lun);
(so remove /block)
/* OK, not . or ..; let's see if it is a SCSI device */ if (len > 8 && block_dirent->d_name[0] == 'b' && block_dirent->d_name[1] == 'l' && block_dirent->d_name[2] == 'o' && block_dirent->d_name[3] == 'c' && block_dirent->d_name[4] == 'k' && block_dirent->d_name[5] == ':' && block_dirent->d_name[6] == 's' && block_dirent->d_name[7] == 'd') { /* looks like a scsi device, smells like scsi device; it must be a scsi device */ dev = (char *) calloc(sizeof(char), len - 5); strncpy(dev, &(block_dirent->d_name[6]), (len - 6));
I guess that can be come a strncmp. And for sake of memory management an if (dev != NULL) would be good too.
Yes, except the problem with your patch is that it will probably break the kernel I made the fixes for originally, which was either 2.6.23 or 2.6.24 (and which had the /sys/class/iscsi_session/session9/device/target9:0:0:0/ 9:0:0:0/block/sdd type paths). We may be able to come up with a hybrid solution that will work on both cases, though. Chris Lalancette
On Wed, 14 May 2008, Chris Lalancette wrote:
Yes, except the problem with your patch is that it will probably break the kernel I made the fixes for originally, which was either 2.6.23 or 2.6.24 (and which had the /sys/class/iscsi_session/session9/device/target9:0:0:0/ 9:0:0:0/block/sdd type paths). We may be able to come up with a hybrid solution that will work on both cases, though.
...are you saying that open-iscsi and the kernel are different trees? As far as I can see the latest kernel for Xen in Gentoo is 2.6.21, is the current Xen code in the vanilla kernel stable (enough)? What do you think about these errors: Link '/dev/disk/by-path/ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-2' does not point to '/dev/sdd', ignoring Link '/dev/disk/by-path/ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-2' does not point to '/dev/sdc', ignoring 1) Lacks a newline 2) In gentoo the pointer is ../../sda or something like it, so it is physically the same device. I don't really know if it has any influence, but it looks dirty to me. Stefan
On Wed, May 14, 2008 at 03:19:58PM +0200, Chris Lalancette wrote:
Stefan de Konink wrote:
Fix: snprintf(sysfs_path, PATH_MAX, "/sys/class/iscsi_session/session%s/device/" "target%d:%d:%d/%d:%d:%d:%d", session, target, channel, id, target, channel, id, lun);
(so remove /block)
/* OK, not . or ..; let's see if it is a SCSI device */ if (len > 8 && block_dirent->d_name[0] == 'b' && block_dirent->d_name[1] == 'l' && block_dirent->d_name[2] == 'o' && block_dirent->d_name[3] == 'c' && block_dirent->d_name[4] == 'k' && block_dirent->d_name[5] == ':' && block_dirent->d_name[6] == 's' && block_dirent->d_name[7] == 'd') { /* looks like a scsi device, smells like scsi device; it must be a scsi device */ dev = (char *) calloc(sizeof(char), len - 5); strncpy(dev, &(block_dirent->d_name[6]), (len - 6));
I guess that can be come a strncmp. And for sake of memory management an if (dev != NULL) would be good too.
Yes, except the problem with your patch is that it will probably break the kernel I made the fixes for originally, which was either 2.6.23 or 2.6.24 (and which had the /sys/class/iscsi_session/session9/device/target9:0:0:0/ 9:0:0:0/block/sdd type paths). We may be able to come up with a hybrid solution that will work on both cases, though.
Before I ACK any patches to the iSCSI driver to deal with other kernel versions I want a documentation patch describing the exact sysfs trees we need to deal with for the particular kernel variants the future code patch is intended to support. This is the only way we stand a chance of being able to maintain this long term. Dan. -- |: Red Hat, Engineering, Boston -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
I think if you can get iscsi_sysfs.c; get_block_dev_from_lun you have the code already to do this all. So basically get openiscsi into a lib with helper functions. ISCSI_LIB_SRCS = util.o io.o auth.o login.o log.o md5.o sha1.o iface.o idbm.o iscsi_sysfs.o To something like libopen-iscsi.so, and link it against libvirt, you have what is required. Stefan
Stefan de Konink wrote:
I think if you can get iscsi_sysfs.c; get_block_dev_from_lun you have the code already to do this all. So basically get openiscsi into a lib with helper functions.
OK. I took a look at this further, and I figured out what is going on here. The original impetus for me to re-write the code in virStorageBackendISCSIMakeLUN() was because I thought the open-iscsi people had changed the output of iscsiadm --mode session -r $session -P 3. However, that's not what actually happened. What actually happened is that the sysfs files that export this information changed somewhere between 2.6.21 and 2.6.24 (which is what I was using at the time). So, the code you point to above in open-iscsi tools *also* doesn't work. That code is expecting the block devices to look like block:sdd, which is what you have in your 2.6.21 kernel; but in later kernels, it no longer looks like that. All that being said, the less we have to depend on parsing, and the more we can poke around sysfs and get values, the better. So in that sense, the change was for the better. I've attached a patch which seems to do the right thing on my 2.6.25 based machine; can you test it on 2.6.21 and see if it works? Chris Lalancette
On Wed, May 21, 2008 at 04:50:12PM +0200, Chris Lalancette wrote:
Stefan de Konink wrote:
I think if you can get iscsi_sysfs.c; get_block_dev_from_lun you have the code already to do this all. So basically get openiscsi into a lib with helper functions.
OK. I took a look at this further, and I figured out what is going on here.
The original impetus for me to re-write the code in virStorageBackendISCSIMakeLUN() was because I thought the open-iscsi people had changed the output of iscsiadm --mode session -r $session -P 3. However, that's not what actually happened. What actually happened is that the sysfs files that export this information changed somewhere between 2.6.21 and 2.6.24 (which is what I was using at the time). So, the code you point to above in open-iscsi tools *also* doesn't work. That code is expecting the block devices to look like block:sdd, which is what you have in your 2.6.21 kernel; but in later kernels, it no longer looks like that.
All that being said, the less we have to depend on parsing, and the more we can poke around sysfs and get values, the better. So in that sense, the change was for the better. I've attached a patch which seems to do the right thing on my 2.6.25 based machine; can you test it on 2.6.21 and see if it works?
Can you provide a documentation patch describing the sysfs layout for the kernels the patch is intended to support, along with captured output from the isciadm tool running on each, for the various commands libvirt needs to run. The only way we are going to avoid breaking this repeatedly is to get serious around documentating the variants we need to support then then testing them. The example output from iscsidam tool then can be used to build a fake iscsiadm tool invoked by the test suites exercising all the new code. Regards, Daniel. -- |: Red Hat, Engineering, Boston -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel P. Berrange wrote:
Can you provide a documentation patch describing the sysfs layout for the kernels the patch is intended to support, along with captured output from the isciadm tool running on each, for the various commands libvirt needs to run.
The only way we are going to avoid breaking this repeatedly is to get serious around documentating the variants we need to support then then testing them. The example output from iscsidam tool then can be used to build a fake iscsiadm tool invoked by the test suites exercising all the new code.
I didn't get all of the information you requested (yet). But I did connect a number of different kernels up to an iSCSI target and look at the listing in /sys/bus/scsi/devices/target:channel:id:lun Kernel iscsi-initiator-utils sysfs 2.6.9 4.0.3.0-6 block/sda1, block/sda2 (note no block/sda!) 2.6.18 6.2.0.868-0.7.el5 block:sda 2.6.21 6.2.0.865-0.2.fc8 block:sda 2.6.25 6.2.0.868-0.7.fc9.i386 block/sda Now, we probably don't care about 2.6.9. So that leaves the other few kernels, which all seem to follow either the block:sda or block/sda pattern. So, at least for now, Stefan can test my patch to see if it helps him. I'll try to collect further information, including the output from iscsiadm. Chris Lalancette
Chris Lalancette schreef:
Stefan de Konink wrote:
I think if you can get iscsi_sysfs.c; get_block_dev_from_lun you have the code already to do this all. So basically get openiscsi into a lib with helper functions.
OK. I took a look at this further, and I figured out what is going on here.
I can confirm this works for me. Stefan
On Wed, May 14, 2008 at 01:57:13PM +0200, Chris Lalancette wrote:
Stefan de Konink wrote:
snprintf(sysfs_path, PATH_MAX, "/sys/class/iscsi_session/session%s/device/" "target%d:%d:%d/%d:%d:%d:%d/block", session, target, channel, id, target, channel, id, lun);
xen01 5:0:0:1 # pwd /sys/class/iscsi_session/session5/device/target5:0:0/5:0:0:1 xen01 5:0:0:1 # ls -l total 0 lrwxrwxrwx 1 root root 0 May 14 13:14 block:sdd -> ../../../../../../block/sdd
Ug. Yeah, that will do it. What kernel version are you running that your sysfs looks like that?
It seems we have been shoved into a corner; we can't rely on the output of the iscsiadm tools, since we have already seen those change between versions. And apparently we can't rely on sysfs paths to stay stable either. Either we need some iscsiadm command with a known, stable output that we can call, or we need some sort of iSCSI API that we can call to get this information. Either way, I think we need help from the iSCSI people, because the current position seem to be untenable.
We have no choice. - No standard iSCSI library exists for us to use - Any iSCSI library would encounter these same problems - IF a library were written it still won't be in any distro we use for a long time So we have no choice but to make our current code work. If someone wants to pursue development of an iSCSI library as a long term solution that's a worthy goal, but for libvirt we need solutions that work today, for existing distros. This comes down to, documenting the variations we have to deal with, figuring out the code to support them, and writing test cases to ensure it works and we don't cause regressions. Dan. -- |: Red Hat, Engineering, Boston -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Wed, 14 May 2008, Daniel P. Berrange wrote:
So we have no choice but to make our current code work. If someone wants to pursue development of an iSCSI library as a long term solution that's a worthy goal, but for libvirt we need solutions that work today, for existing distros.
Why not ask the open-iscsi community to export their code in a library? Stefan
participants (4)
-
Chris Lalancette -
Daniel P. Berrange -
Mark Dehus -
Stefan de Konink