[libvirt] Share storage using iscsi [EDIT as I cut off part]
Hi, I'm trying to setup pool of machines (nodes) for virtual machines hosting and I got few question about shared storage. My main requirements are: 1. central management - I've got simple python app that stores information about all virtual machines and all nodes, this app needs to be able to manage volumes using libvirt API, so I need libvirt volume pools 2. live migration - I got shared storage with HA, I want to use it also for live migration in case one of nodes is dying or if I want to do some load balancing Right now I'm thinking about 2 machines with disks synchronized using drdb, both acting as a identical iscsi targets, iscsi HA will be provided by heartbeat. So I will end up with virtual IP pointing to working iscsi target, drdb should keep storage is sync. But: 1. I can't just use single iscsi LUN and export it as libvirt storage pool to each node, because no pool type would work that way, right? 2. http://libvirt.org/storage.html section "iSCSI volume pools" says: "Volumes must be pre-allocated on the iSCSI server, and cannot be created via the libvirt APIs." So even if I got one LUN per node and set it as iscsi volume pool I would need to create each volume on iscsi target. Libvirt can't manage volumes in such pool, it can only assign already created volumes to virtual machines, right? 3. So maybe my storage could be setup as LVM volume group and this lvm group would be managed as libvirt lvm volume pool on master (from heartbeat POV) iscsi target. I would create one logical volume per virtual machine, export this volume as a separate iscsi LUN, and use this LUN as iscsi volume for virtual machine. To create new virtual machine I would: a) create lvm volume on iscsi target using libvirt b) export this volume using iscsi c) define virtual machine with this volume But can I create iscsi volume without pool? Does it makes any sense? Are there better ways to *manage* volumes for virtual machines using iscsi? Thanks for any tips. Łukasz Mierzwa
Łukasz Mierzwa wrote: Hi Łukasz, I will try to answer your questions inline.
Hi,
I'm trying to setup pool of machines (nodes) for virtual machines hosting and I got few question about shared storage. My main requirements are: 1. central management - I've got simple python app that stores information about all virtual machines and all nodes, this app needs to be able to manage volumes using libvirt API, so I need libvirt volume pools
Storage pools will let you manage iSCSI LUNs as volumes. When you start the iSCSI pool, the node will login to the iSCSI target and libvirt will create a volume in the pool for each LUN on that target.
2. live migration - I got shared storage with HA, I want to use it also for live migration in case one of nodes is dying or if I want to do some load balancing
By HA you mean you have two targets presenting the same LUNs, or do you mean something else?
Right now I'm thinking about 2 machines with disks synchronized using drdb, both acting as a identical iscsi targets, iscsi HA will be provided by heartbeat. So I will end up with virtual IP pointing to working iscsi target, drdb should keep storage is sync. But:
1. I can't just use single iscsi LUN and export it as libvirt storage pool to each node, because no pool type would work that way, right?
You could create a logical pool on the iSCSI LUN and libvirt can create and destroy volumes in the logical pool, which I think is what you're saying below.
2. http://libvirt.org/storage.html section "iSCSI volume pools" says: "Volumes must be pre-allocated on the iSCSI server, and cannot be created via the libvirt APIs." So even if I got one LUN per node and set it as iscsi volume pool I would need to create each volume on iscsi target. Libvirt can't manage volumes in such pool, it can only assign already created volumes to virtual machines, right?
What that means is that libvirt doesn't know how to tell an iSCSI array to create a new LUN. You have to use whatever admin tools you use to manage the array to create new LUNs, but then libvirt will discover them with a pool refresh and you can assign them to guests.
3. So maybe my storage could be setup as LVM volume group and this lvm group would be managed as libvirt lvm volume pool on master (from heartbeat POV) iscsi target. I would create one logical volume per virtual machine, export this volume as a separate iscsi LUN, and use this LUN as iscsi volume for virtual machine. To create new virtual machine I would: a) create lvm volume on iscsi target using libvirt b) export this volume using iscsi c) define virtual machine with this volume But can I create iscsi volume without pool?
I think I'm following you, but let me see if I can restate what you're saying: a) With your array's management tools, not with with libvirt, create a single iSCSI LUN (or several LUNs) on a target on your array. b) With libvirt, create an iSCSI storage pool for that iSCSI target. c) With libvirt, create a logical storage pool on that LUN, which creates a LVM group on that LUN. d) With libvirt, you create (and destroy) logical volumes in the LVM group and assign them to guests.
Does it makes any sense? Are there better ways to *manage* volumes for virtual machines using iscsi?
And just to make sure I'm clear on what you mean by manage, you mean create and destroy, right? Dave
2009/8/14 Dave Allan <dallan@redhat.com>
Łukasz Mierzwa wrote:
Hi Łukasz,
I will try to answer your questions inline.
Hi,
I'm trying to setup pool of machines (nodes) for virtual machines hosting and I got few question about shared storage. My main requirements are: 1. central management - I've got simple python app that stores information about all virtual machines and all nodes, this app needs to be able to manage volumes using libvirt API, so I need libvirt volume pools
Storage pools will let you manage iSCSI LUNs as volumes. When you start the iSCSI pool, the node will login to the iSCSI target and libvirt will create a volume in the pool for each LUN on that target.
2. live migration - I got shared storage with HA, I want to use it also
for live migration in case one of nodes is dying or if I want to do some load balancing
By HA you mean you have two targets presenting the same LUNs, or do you mean something else?
Yes, both targets have the same LUNs and data stored in those LUNs is synchronized using drdb. Like described in this howto https://help.ubuntu.com/community/HighlyAvailableiSCSITarget
Right now I'm thinking about 2 machines with disks synchronized using
drdb, both acting as a identical iscsi targets, iscsi HA will be provided by heartbeat. So I will end up with virtual IP pointing to working iscsi target, drdb should keep storage is sync. But:
1. I can't just use single iscsi LUN and export it as libvirt storage pool to each node, because no pool type would work that way, right?
You could create a logical pool on the iSCSI LUN and libvirt can create and destroy volumes in the logical pool, which I think is what you're saying below.
2. http://libvirt.org/storage.html section "iSCSI volume pools" says:
"Volumes must be pre-allocated on the iSCSI server, and cannot be created via the libvirt APIs." So even if I got one LUN per node and set it as iscsi volume pool I would need to create each volume on iscsi target. Libvirt can't manage volumes in such pool, it can only assign already created volumes to virtual machines, right?
What that means is that libvirt doesn't know how to tell an iSCSI array to create a new LUN. You have to use whatever admin tools you use to manage the array to create new LUNs, but then libvirt will discover them with a pool refresh and you can assign them to guests.
3. So maybe my storage could be setup as LVM volume group and this lvm
group would be managed as libvirt lvm volume pool on master (from heartbeat POV) iscsi target. I would create one logical volume per virtual machine, export this volume as a separate iscsi LUN, and use this LUN as iscsi volume for virtual machine. To create new virtual machine I would: a) create lvm volume on iscsi target using libvirt b) export this volume using iscsi c) define virtual machine with this volume But can I create iscsi volume without pool?
I think I'm following you, but let me see if I can restate what you're saying:
a) With your array's management tools, not with with libvirt, create a single iSCSI LUN (or several LUNs) on a target on your array. b) With libvirt, create an iSCSI storage pool for that iSCSI target. c) With libvirt, create a logical storage pool on that LUN, which creates a LVM group on that LUN. d) With libvirt, you create (and destroy) logical volumes in the LVM group and assign them to guests.
Does it makes any sense? Are there better ways to *manage* volumes for
virtual machines using iscsi?
And just to make sure I'm clear on what you mean by manage, you mean create and destroy, right?
Create, destroy and possibly resize (which means grow as shrinking filesystems is complex and more dangerous) virtual machine volumes.
Dave
Let me make that little more clear: 1. Lets assume I have one storage node that is acting as a iscsi target. 2. This storage node has one big block device (RAID of some kind) that will be used for storing virtual machines volumes 3. I have multiple nodes for hosting virtual machines. 4. Each of those hosting nodes needs to be able to access every volume stored on the storage node (for live migration) using iscsi. I'm looking for proper way to do this. Right now I'm thinking about this setup: 1. On the storage node I will create lvm volume group using my RAID. 2. Each virtual machine volume is created as lvm logical volume, it doesn't matter how this volume is created (by hand, using management app or any other way) but it always happens on the storage node. I could export whole RAID as a single LUN, create lvm volume group on top of that, but to be able to work with this lvm volume group from every hosting node at the same time I would need to use clustered lvm (or something like that) and that would add some complexity. 3. Each lvm logical volume is exported as unique LUN (also out of libvirt scope) 4. iscsi LUNs are discovered using libvirt iscsi volume pool, and assigned to right virtual machine. Łukasz Mierzwa
Łukasz Mierzwa wrote:
2009/8/14 Dave Allan <dallan@redhat.com <mailto:dallan@redhat.com>>
Łukasz Mierzwa wrote:
Hi Łukasz,
I will try to answer your questions inline.
Hi,
I'm trying to setup pool of machines (nodes) for virtual machines hosting and I got few question about shared storage. My main requirements are: 1. central management - I've got simple python app that stores information about all virtual machines and all nodes, this app needs to be able to manage volumes using libvirt API, so I need libvirt volume pools
Storage pools will let you manage iSCSI LUNs as volumes. When you start the iSCSI pool, the node will login to the iSCSI target and libvirt will create a volume in the pool for each LUN on that target.
2. live migration - I got shared storage with HA, I want to use it also for live migration in case one of nodes is dying or if I want to do some load balancing
By HA you mean you have two targets presenting the same LUNs, or do you mean something else?
Yes, both targets have the same LUNs and data stored in those LUNs is synchronized using drdb. Like described in this howto https://help.ubuntu.com/community/HighlyAvailableiSCSITarget
Right now I'm thinking about 2 machines with disks synchronized using drdb, both acting as a identical iscsi targets, iscsi HA will be provided by heartbeat. So I will end up with virtual IP pointing to working iscsi target, drdb should keep storage is sync. But:
1. I can't just use single iscsi LUN and export it as libvirt storage pool to each node, because no pool type would work that way, right?
You could create a logical pool on the iSCSI LUN and libvirt can create and destroy volumes in the logical pool, which I think is what you're saying below.
2. http://libvirt.org/storage.html section "iSCSI volume pools" says: "Volumes must be pre-allocated on the iSCSI server, and cannot be created via the libvirt APIs." So even if I got one LUN per node and set it as iscsi volume pool I would need to create each volume on iscsi target. Libvirt can't manage volumes in such pool, it can only assign already created volumes to virtual machines, right?
What that means is that libvirt doesn't know how to tell an iSCSI array to create a new LUN. You have to use whatever admin tools you use to manage the array to create new LUNs, but then libvirt will discover them with a pool refresh and you can assign them to guests.
3. So maybe my storage could be setup as LVM volume group and this lvm group would be managed as libvirt lvm volume pool on master (from heartbeat POV) iscsi target. I would create one logical volume per virtual machine, export this volume as a separate iscsi LUN, and use this LUN as iscsi volume for virtual machine. To create new virtual machine I would: a) create lvm volume on iscsi target using libvirt b) export this volume using iscsi c) define virtual machine with this volume But can I create iscsi volume without pool?
I think I'm following you, but let me see if I can restate what you're saying:
a) With your array's management tools, not with with libvirt, create a single iSCSI LUN (or several LUNs) on a target on your array. b) With libvirt, create an iSCSI storage pool for that iSCSI target. c) With libvirt, create a logical storage pool on that LUN, which creates a LVM group on that LUN. d) With libvirt, you create (and destroy) logical volumes in the LVM group and assign them to guests.
Does it makes any sense? Are there better ways to *manage* volumes for virtual machines using iscsi?
And just to make sure I'm clear on what you mean by manage, you mean create and destroy, right?
Create, destroy and possibly resize (which means grow as shrinking filesystems is complex and more dangerous) virtual machine volumes.
Dave
Let me make that little more clear: 1. Lets assume I have one storage node that is acting as a iscsi target. 2. This storage node has one big block device (RAID of some kind) that will be used for storing virtual machines volumes 3. I have multiple nodes for hosting virtual machines. 4. Each of those hosting nodes needs to be able to access every volume stored on the storage node (for live migration) using iscsi.
I'm looking for proper way to do this. Right now I'm thinking about this setup: 1. On the storage node I will create lvm volume group using my RAID. 2. Each virtual machine volume is created as lvm logical volume, it doesn't matter how this volume is created (by hand, using management app or any other way) but it always happens on the storage node. I could export whole RAID as a single LUN, create lvm volume group on top of that, but to be able to work with this lvm volume group from every hosting node at the same time I would need to use clustered lvm (or something like that) and that would add some complexity. 3. Each lvm logical volume is exported as unique LUN (also out of libvirt scope) 4. iscsi LUNs are discovered using libvirt iscsi volume pool, and assigned to right virtual machine.
I see what you're proposing, and you're correct. If you confine the visibility of the LVs to the iSCSI target and expose each one as a separate iSCSI LUN, you don't need clvm on the nodes. That will work fine, and libvirt will handle the discovery through a refresh of the volume pool. As you point out, libvirt won't handle telling the iSCSI target to create the new LUNs. If you go the route of exposing the VG to the nodes, you would need clvm on the nodes. As a side note, I don't object to creating functionality to allow libvirt to manage iSCSI targets, extending the volume create/delete code to interface with the target. However, I don't want to try to create code to manage every possible iSCSI target. If you wanted to submit a patch to libvirt to do the kind of target management you're looking for, I would support that effort, and I'd be happy to work with you to set up a framework for plugging in code to manage whatever arrays anybody wanted to support. You're not the only person to want that functionality, and if it's done in a well-defined way, I think it would be a good thing. Others on the list might have different opinions. Dave
On Mon, Aug 17, 2009 at 03:03:12PM -0400, Dave Allan wrote:
As a side note, I don't object to creating functionality to allow libvirt to manage iSCSI targets, extending the volume create/delete code to interface with the target. However, I don't want to try to create code to manage every possible iSCSI target. If you wanted to submit a patch to libvirt to do the kind of target management you're looking for, I would support that effort, and I'd be happy to work with you to set up a framework for plugging in code to manage whatever arrays anybody wanted to support. You're not the only person to want that functionality, and if it's done in a well-defined way, I think it would be a good thing. Others on the list might have different opinions.
That's quite an interesting idea, but it doesn't neccessarily have to be done inside libvirt itself. For example, we could add lifecycle events to the libvirt storage APIs, so that a application can receive a notification if a pool is created/ deleted/started/stopped, and also notifications when volumes are created or deleted within a pool. With that, you could easily write a standalone daemon that listened to the events and automatically updated a iSCSI target's LUNs when volumes in a particular pool are added/deleted. Or created/delete targets automatically when libvirt pools are added/deleted. I think this'd be quite a nice little daemon/service to have, providing a simple iSCSI target service you can use in conjunction with libvirt. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Dnia poniedziałek 17 sierpień 2009 o 21:03:12 Dave Allan napisał(a):
Łukasz Mierzwa wrote: [...]
Let me make that little more clear: 1. Lets assume I have one storage node that is acting as a iscsi target. 2. This storage node has one big block device (RAID of some kind) that will be used for storing virtual machines volumes 3. I have multiple nodes for hosting virtual machines. 4. Each of those hosting nodes needs to be able to access every volume stored on the storage node (for live migration) using iscsi.
I'm looking for proper way to do this. Right now I'm thinking about this setup: 1. On the storage node I will create lvm volume group using my RAID. 2. Each virtual machine volume is created as lvm logical volume, it doesn't matter how this volume is created (by hand, using management app or any other way) but it always happens on the storage node. I could export whole RAID as a single LUN, create lvm volume group on top of that, but to be able to work with this lvm volume group from every hosting node at the same time I would need to use clustered lvm (or something like that) and that would add some complexity. 3. Each lvm logical volume is exported as unique LUN (also out of libvirt scope) 4. iscsi LUNs are discovered using libvirt iscsi volume pool, and assigned to right virtual machine.
I see what you're proposing, and you're correct. If you confine the visibility of the LVs to the iSCSI target and expose each one as a separate iSCSI LUN, you don't need clvm on the nodes. That will work fine, and libvirt will handle the discovery through a refresh of the volume pool. As you point out, libvirt won't handle telling the iSCSI target to create the new LUNs.
If you go the route of exposing the VG to the nodes, you would need clvm on the nodes.
As a side note, I don't object to creating functionality to allow libvirt to manage iSCSI targets, extending the volume create/delete code to interface with the target. However, I don't want to try to create code to manage every possible iSCSI target. If you wanted to submit a patch to libvirt to do the kind of target management you're looking for, I would support that effort, and I'd be happy to work with you to set up a framework for plugging in code to manage whatever arrays anybody wanted to support. You're not the only person to want that functionality, and if it's done in a well-defined way, I think it would be a good thing. Others on the list might have different opinions.
That would be great but unfortunately I have not much experience with C/C++ and I'm afraid that this would take a while for me to achieve. I'm more of sysadmin with some python skills than a programmer. Also my spare time is limited as I'm working on a hobby project (http://github.com/prymitive/pumpkin). Łukasz Mierzwa
participants (3)
-
Daniel P. Berrange -
Dave Allan -
Łukasz Mierzwa