[libvirt-users] FYI: Notes on setting up KVM guests using iSCSI
DV suggested that we document some libvirt setups using shared storage. I'm not a fan of NFS, so I wrote some blog posts on how to use iSCSI in the context of libvirt + KVM. There is of course more than one way todo things, so I've outlined a couple of different options. One completely manual command line approach using tgtadm on the iSCSI server: http://berrange.com/posts/2010/05/05/provisioning-kvm-virtual-machines-on-is... ..and virsh/virt-install on the KVM host http://berrange.com/posts/2010/05/05/provisioning-kvm-virtual-machines-on-is... Then one completely GUI based approach, using a QNAP NAS as the iSCSI server http://berrange.com/posts/2010/05/04/provisioning-kvm-virtual-machines-on-is... ..and a virt-manager on the KVM host: http://berrange.com/posts/2010/05/04/provisioning-kvm-virtual-machines-on-is... You can of course mix+match, doing the iSCSI server the manual way and the libvirt bits the GUI way and vica-verca. Any other commercial iSCSI server will have the same concepts so the libvirt instructions apply in all likely scenarios. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 05/05/2010 07:23 AM, Daniel P. Berrange wrote:
DV suggested that we document some libvirt setups using shared storage. I'm not a fan of NFS, so I wrote some blog posts on how to use iSCSI in the context of libvirt + KVM.
Thank you for creating and sharing these.
http://berrange.com/posts/2010/05/05/provisioning-kvm-virtual-machines-on-is...
I have a question about this example, specifically the shareable permission option and tag: (perm=sh, <shareable/>) What does that do, exactly? Is it simply the required r/w option to prevent corruption of VMs when using shared storage, or is it needed only if the disk will be actively by more than one VM? (That 2nd idea doesn't seem quite right to me, I would expect to have to use some kind of locking technology to prevent it two or more VMs trying to write to the same disk at the same time). Thanks, Tom
On Wed, May 05, 2010 at 08:09:50AM -0400, Tom Georgoulias wrote:
On 05/05/2010 07:23 AM, Daniel P. Berrange wrote:
DV suggested that we document some libvirt setups using shared storage. I'm not a fan of NFS, so I wrote some blog posts on how to use iSCSI in the context of libvirt + KVM.
Thank you for creating and sharing these.
http://berrange.com/posts/2010/05/05/provisioning-kvm-virtual-machines-on-is...
I have a question about this example, specifically the shareable permission option and tag: (perm=sh, <shareable/>)
What does that do, exactly? Is it simply the required r/w option to prevent corruption of VMs when using shared storage, or is it needed only if the disk will be actively by more than one VM? (That 2nd idea doesn't seem quite right to me, I would expect to have to use some kind of locking technology to prevent it two or more VMs trying to write to the same disk at the same time).
The <shareable/> flag tells libvirt that the disk will be *concurrently* accessed by multiple VMs. This means that the hypervisor will not try to take an exclusive lock on the disk, that all I/O caching is disabled, and any SELinux labelling allows use by all VMs. NB: if you are marking a disk as <sharable/> you need to know what you are doing in the guest OS. You would typically use a cluster filesystem on the disk like GFS or OCFS. Attempts to use a normal FS like ext3 from multiple VMs at once is not healthy for your data :-) Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel, Thank you Very much for you efforts. I'll be studying these with interest. I also plan on replying to the previous response to my request for help on shard storage later today. I'm still trying to run some tests to see if I understand how the Pools vs Volumes work with libvirt and iSCSI. Thanks again! David. On Wed, 5 May 2010, Daniel P. Berrange wrote:
DV suggested that we document some libvirt setups using shared storage. I'm not a fan of NFS, so I wrote some blog posts on how to use iSCSI in the context of libvirt + KVM.
There is of course more than one way todo things, so I've outlined a couple of different options. One completely manual command line approach using tgtadm on the iSCSI server:
http://berrange.com/posts/2010/05/05/provisioning-kvm-virtual-machines-on-is...
..and virsh/virt-install on the KVM host
http://berrange.com/posts/2010/05/05/provisioning-kvm-virtual-machines-on-is...
Then one completely GUI based approach, using a QNAP NAS as the iSCSI server
http://berrange.com/posts/2010/05/04/provisioning-kvm-virtual-machines-on-is...
..and a virt-manager on the KVM host:
http://berrange.com/posts/2010/05/04/provisioning-kvm-virtual-machines-on-is...
You can of course mix+match, doing the iSCSI server the manual way and the libvirt bits the GUI way and vica-verca. Any other commercial iSCSI server will have the same concepts so the libvirt instructions apply in all likely scenarios.
Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On Wed, May 05, 2010 at 08:28:15AM -0500, David Ehle wrote:
Daniel,
Thank you Very much for you efforts. I'll be studying these with interest.
I also plan on replying to the previous response to my request for help on shard storage later today. I'm still trying to run some tests to see if I understand how the Pools vs Volumes work with libvirt and iSCSI.
For iSCSI - Pool == A single target - Volume == A LUN within a target So, if your iSCSI server exports many iSCSI targets, you'd setup a separate pool for each target you want to access from libvirt. This post covers iSCSI pool setup at a low level: http://berrange.com/posts/2010/05/05/provisioning-kvm-virtual-machines-on-is... Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel, OK so I think I get it now. While for other storage backends one pool can hold multiple volumes, when using iSCIS there is alwasy a 1 to 1 relationship for pool to volume? Each pool contains exactly one volume? David. On Wed, 5 May 2010, Daniel P. Berrange wrote:
On Wed, May 05, 2010 at 08:28:15AM -0500, David Ehle wrote:
Daniel,
Thank you Very much for you efforts. I'll be studying these with interest.
I also plan on replying to the previous response to my request for help on shard storage later today. I'm still trying to run some tests to see if I understand how the Pools vs Volumes work with libvirt and iSCSI.
For iSCSI
- Pool == A single target - Volume == A LUN within a target
So, if your iSCSI server exports many iSCSI targets, you'd setup a separate pool for each target you want to access from libvirt. This post covers iSCSI pool setup at a low level:
http://berrange.com/posts/2010/05/05/provisioning-kvm-virtual-machines-on-is...
Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Wed, May 05, 2010 at 08:50:57AM -0500, David Ehle wrote:
Daniel,
OK so I think I get it now. While for other storage backends one pool can hold multiple volumes, when using iSCIS there is alwasy a 1 to 1 relationship for pool to volume? Each pool contains exactly one volume?
No, each pool can contain multiple volumes. If you have created a iSCSI target and given it multiple LUNs, then you'll see multiple volumes in libvirt. See the virsh vol-list output in this example:
http://berrange.com/posts/2010/05/05/provisioning-kvm-virtual-machines-on-is...
for an iSCSI target with 2 LUNs Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel, Thank you. I was misunderstanding how iscsi iSCSI Enterprise Target Daemon /ietd.conf work. I was setting up multiple targets with only one LUN each, and thinking I was creating multiple LUNs on the same target. So right way to set up multiple LUNS for one target under Ubuntu Karmic using iSCSI Enterprise Target via ietd.conf: Target iqn.2010-04.gov.blah.blah.blah.keeper:vmstorage <SNIP> # Logical Unit definition # You must define one logical unit at least. # Block devices, regular files, LVM, and RAID can be offered # to the initiators as a block device. Lun 0 Path=/dev/vg0/iscsi_pool,Type=fileio Lun 1 Path=/dev/vg0/test_pool,Type=fileio # Alias name for this target Alias VMStoragePool <SNIP> <NOT> Target iqn.2010-04.gov.blah.blah.blah.keeper:vmstorage.lun0 Lun 0 Path=/dev/vg0/iscsi_pool,Type=fileio ALIAS LUN0 Target iqn.2010-04.gov.blah.blah.blah.keeper:vmstorage.lun1 Lun 1 Path=/dev/vg0/test_pool,Type=fileio ALIAS LUN1 </NOT> To add additional volumes, additional LUNs will need to be created on the TARGET side, and then exported via iSCSI After correcting my Target side configuration, I was able to create an iSCSI pool through virt-manager, and see the two LUNs as available volumes. On Wed, 5 May 2010, Daniel P. Berrange wrote:
On Wed, May 05, 2010 at 08:50:57AM -0500, David Ehle wrote:
Daniel,
OK so I think I get it now. While for other storage backends one pool can hold multiple volumes, when using iSCIS there is alwasy a 1 to 1 relationship for pool to volume? Each pool contains exactly one volume?
No, each pool can contain multiple volumes. If you have created a iSCSI target and given it multiple LUNs, then you'll see multiple volumes in libvirt.
See the virsh vol-list output in this example:
http://berrange.com/posts/2010/05/05/provisioning-kvm-virtual-machines-on-is...
for an iSCSI target with 2 LUNs
Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
participants (3)
-
Daniel P. Berrange -
David Ehle -
Tom Georgoulias