[libvirt] Associate a LUN as disk with WWNN/WWPN
Hi, all This is a rough thought about how to allow the user to associate a LUN as disk with WWNN/WWPN. Never mind if some thought is complete wrong, I'm complete new to this area. :-) The initial idea was to introduce new XML for the vHBA which the LUN which be assisned to in a domain XML, just like comment #2 of RHBZ# 543037: <quote> AFAIU libvirt needs a way to: - Associate a virtual adapter WWN with a VM (in the VM xml) - Learn to start a virtual adapter when the VM is started, and destroy the adapter when the VM is stopped. - Possibly a way to associate a WWN with a scsi pool, to start / stop a virtual adapter with the pool. </quote> But afer thinking more, I'd think it might be not good idea: As far as I could understand, the requirement of the BZ wants a way to create/migrate a guest with NPIV. I'm goint to talk create first,and migration then. === Creation If we try to support this all inside libvirt, it means we need to new XML for indicates the WWNN/WWPN of the vHBA or Physical HBA, this is fine we can do it, though it's a bit annoyed where to add it. The biggest problem is there is a race here, we can't known the path of the LUN unless the storage admin assigned (zone / mask, will already use assign for brevity) a LUN to the (v)HBA, and it appears on host successfully. However, domain define/create doesn't allow the disk source path be empty, except it's CDROM, or Floppy. The works which need to be done before domain define should be: - If the user wants to create new vHBA. 1) auto-generate /or specify WWNN/WWPN (with Qumranet's OUI: 00-1A-4A) 2) allows to select fabric WWN, (to guide the choice of which FC adapter to select for the Vport creation, assuming different adapters connect to different fabrics) 3) create the vHBA with virNodeDeviceCreateXML API 4) Ask / prompt the user to ask the storage admin assigns the LUN to the vHBA. 5) Waits for the LUN shows up on host (perhaps via storage pool API) 6) Use the LUN's path (e.g. /dev/disk/by-id/scsi-3600c0ff000d7a2a5d163ff4490100000000-part1) as the disk source path. - If the user wants to select a existed (v)HBA. 1), 2) and 3) are ommited 3) and 4) might be also ommited if there is a LUN already assigned to the selected (v)HBA. As we can see, there are lots of interaction. If we simply wants to do all these work inside libvirt, we lose the user friendly interaction, and just makes our life harder. So, I would these work should be done in external / upper apps, e.g. virt-manager, or Boxes in future. What libvirt needs to do (as far as I could get): 1) support the fabric WWN in node device XML 2) may be we can do auto-generating the WWNN/WWPN inside libvirt if they are not specified. But from a user's p.o.v, I'd see it is in UI, but not in codes silently. 3) perhaps provides methods to get the (v)HBA's fabric's WWN? === Migration If we do things as explained above for domain's creation, the disk XML will be like: <disk type='disk' device='disk'> <driver name='qemu' type='raw'/> <source file='/dev/disk/by-id/scsi-3600c0ff000d7a2a5d163ff4490100000000-part1/'> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03 function='0x0'/> </disk> It's just similiar with disk which uses a normal disk image. So we needs to something to indentify it's special, it's a LUN assigned to a (v)HBA. Ways to do that in my mind are: 1) If we could find the WWNN/WWPN/fabricwwn by the LUN's path, (I'm not sure if there is a good way yet), things will be much simpler, what we need might be just a simple flag to tell the disk source path is LUN assigned to (v)HBA. e.g. <source file='$path_of_lun' type='fc_hba'/> 2) if there is no way to get the WWNN/WWPN/fabricwwn by LUN's path, we might need the whole config of (v)HBA. e.g. <souce protocol='fc' file='$path_of_lun'> <hba> <wwnn>$wwnn</wwnn> <wwpn>$wwpn</wwpn> <fabricwwn>$fabricwwn</fabricwwn> </hba> </source> And then in virt-manager or Boxes, it needs to detect if there is disk use FC LUN, if it's it should teach dest libvirt to create another vHBA, and assign the same LUN to it. Then migration could do just as normal. But problem of doing like so is the migration won't work anyway with virsh. Hmmm, /me thinking if it will be disireable to introduce hook support for migration. Things will be much simpler if we support hooks? Thoughts and Opinions? Regards, Osier
On Fri, Nov 25, 2011 at 04:55:02PM +0800, Osier Yang wrote:
<quote> AFAIU libvirt needs a way to:
- Associate a virtual adapter WWN with a VM (in the VM xml) - Learn to start a virtual adapter when the VM is started, and destroy the adapter when the VM is stopped. - Possibly a way to associate a WWN with a scsi pool, to start / stop a virtual adapter with the pool. </quote>
But afer thinking more, I'd think it might be not good idea:
As far as I could understand, the requirement of the BZ wants a way to create/migrate a guest with NPIV. I'm goint to talk create first,and migration then.
The desire to assocaite WWNN/WWPN with a VM is just one part of a more general need to expand libvirt's SCSI support. Paulo started a design thread on the subject a month or so back which sort of converged into agreement: https://www.redhat.com/archives/libvir-list/2011-October/msg01253.html Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 2011年11月25日 18:28, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 04:55:02PM +0800, Osier Yang wrote:
<quote> AFAIU libvirt needs a way to:
- Associate a virtual adapter WWN with a VM (in the VM xml) - Learn to start a virtual adapter when the VM is started, and destroy the adapter when the VM is stopped. - Possibly a way to associate a WWN with a scsi pool, to start / stop a virtual adapter with the pool. </quote>
But afer thinking more, I'd think it might be not good idea:
As far as I could understand, the requirement of the BZ wants a way to create/migrate a guest with NPIV. I'm goint to talk create first,and migration then.
The desire to assocaite WWNN/WWPN with a VM is just one part of a more general need to expand libvirt's SCSI support. Paulo started a design thread on the subject a month or so back which sort of converged into agreement:
https://www.redhat.com/archives/libvir-list/2011-October/msg01253.html
Yes, I read this thread before, but it looks to me Paolo was talking about LUN, scsi host, and vHBA passthrough. It's a bit different with what I'm trying to resolve (There is no passthrough here, but about how to design a good workflow between virt-manager / Boxes and libvirt for using (creating and migration ) a FC LUN as a normal disk). The useful thing in the discussion of the thread may be define the (v)HBA as a controller though. Or I misunderstood something? Thanks Osier
On Fri, Nov 25, 2011 at 06:42:42PM +0800, Osier Yang wrote:
On 2011年11月25日 18:28, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 04:55:02PM +0800, Osier Yang wrote:
<quote> AFAIU libvirt needs a way to:
- Associate a virtual adapter WWN with a VM (in the VM xml) - Learn to start a virtual adapter when the VM is started, and destroy the adapter when the VM is stopped. - Possibly a way to associate a WWN with a scsi pool, to start / stop a virtual adapter with the pool. </quote>
But afer thinking more, I'd think it might be not good idea:
As far as I could understand, the requirement of the BZ wants a way to create/migrate a guest with NPIV. I'm goint to talk create first,and migration then.
The desire to assocaite WWNN/WWPN with a VM is just one part of a more general need to expand libvirt's SCSI support. Paulo started a design thread on the subject a month or so back which sort of converged into agreement:
https://www.redhat.com/archives/libvir-list/2011-October/msg01253.html
Yes, I read this thread before, but it looks to me Paolo was talking about LUN, scsi host, and vHBA passthrough. It's a bit different with what I'm trying to resolve (There is no passthrough here, but about how to design a good workflow between virt-manager / Boxes and libvirt for using (creating and migration ) a FC LUN as a normal disk). The useful thing in the discussion of the thread may be define the (v)HBA as a controller though.
Or I misunderstood something?
I've found that when people generally talk about associating iSCSI LUNs with a guest, they have always been expecting SCSI LUN passthrough, or passthrough of the entire iSCSI vHBA. If we're considering a non-passthrough use case, then IMHO the problem should be generalized to How do we associated a storage volume with a VM ? ie not something that is specific to (i)SCSI. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 2011年11月25日 18:49, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 06:42:42PM +0800, Osier Yang wrote:
On 2011年11月25日 18:28, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 04:55:02PM +0800, Osier Yang wrote:
<quote> AFAIU libvirt needs a way to:
- Associate a virtual adapter WWN with a VM (in the VM xml) - Learn to start a virtual adapter when the VM is started, and destroy the adapter when the VM is stopped. - Possibly a way to associate a WWN with a scsi pool, to start / stop a virtual adapter with the pool. </quote>
But afer thinking more, I'd think it might be not good idea:
As far as I could understand, the requirement of the BZ wants a way to create/migrate a guest with NPIV. I'm goint to talk create first,and migration then.
The desire to assocaite WWNN/WWPN with a VM is just one part of a more general need to expand libvirt's SCSI support. Paulo started a design thread on the subject a month or so back which sort of converged into agreement:
https://www.redhat.com/archives/libvir-list/2011-October/msg01253.html
Yes, I read this thread before, but it looks to me Paolo was talking about LUN, scsi host, and vHBA passthrough. It's a bit different with what I'm trying to resolve (There is no passthrough here, but about how to design a good workflow between virt-manager / Boxes and libvirt for using (creating and migration ) a FC LUN as a normal disk). The useful thing in the discussion of the thread may be define the (v)HBA as a controller though.
Or I misunderstood something?
I've found that when people generally talk about associating iSCSI LUNs with a guest, they have always been expecting SCSI LUN passthrough, or passthrough of the entire iSCSI vHBA.
If we're considering a non-passthrough use case, then IMHO the problem should be generalized to
How do we associated a storage volume with a VM ?
ie not something that is specific to (i)SCSI.
Agreed, but as far as I understand about the storage, doesn't only iSCSI need specific setting before use? e.g. For a LVM volume, it doesn't need things like vHBA creating, zone & mask the LUN with storage management tools. Thanks Osier
On Fri, Nov 25, 2011 at 10:49:29AM +0000, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 06:42:42PM +0800, Osier Yang wrote:
On 2011年11月25日 18:28, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 04:55:02PM +0800, Osier Yang wrote:
<quote> AFAIU libvirt needs a way to:
- Associate a virtual adapter WWN with a VM (in the VM xml) - Learn to start a virtual adapter when the VM is started, and destroy the adapter when the VM is stopped. - Possibly a way to associate a WWN with a scsi pool, to start / stop a virtual adapter with the pool. </quote>
But afer thinking more, I'd think it might be not good idea:
As far as I could understand, the requirement of the BZ wants a way to create/migrate a guest with NPIV. I'm goint to talk create first,and migration then.
The desire to assocaite WWNN/WWPN with a VM is just one part of a more general need to expand libvirt's SCSI support. Paulo started a design thread on the subject a month or so back which sort of converged into agreement:
https://www.redhat.com/archives/libvir-list/2011-October/msg01253.html
Yes, I read this thread before, but it looks to me Paolo was talking about LUN, scsi host, and vHBA passthrough. It's a bit different with what I'm trying to resolve (There is no passthrough here, but about how to design a good workflow between virt-manager / Boxes and libvirt for using (creating and migration ) a FC LUN as a normal disk). The useful thing in the discussion of the thread may be define the (v)HBA as a controller though.
Or I misunderstood something?
I've found that when people generally talk about associating iSCSI LUNs with a guest, they have always been expecting SCSI LUN passthrough, or passthrough of the entire iSCSI vHBA.
Agreed, and IMO we should implement that also, but I think that work is divisible from the non-passthrough case, which is what we're considering.
If we're considering a non-passthrough use case, then IMHO the problem should be generalized to
How do we associated a storage volume with a VM ?
ie not something that is specific to (i)SCSI.
I like the idea of associating a storage volume with a particular block device on a VM. So, what's your vision for how a VM would use storage that's visible to a virtual WWN? It sounds like you're saying that we should extend SCSI pools to take WWNN, WWPN and fabric name, and then instantiate the vHBA when the pool is started. We'd then extend the domain disk XML to take a pool/volume definition. When the VM is started, we'd check to see if the pool was active, if not, try to start it, and use the resulting volume as described in the domain XML. We'd have to manage the wait for the volumes to show up, but we have to contend with that regardless. That's kind of a nice option in this use case as it would allow administrators to bring the pool up ahead of domain start if desired. Is that right? Dave
On Mon, Nov 28, 2011 at 09:01:45AM -0500, Dave Allan wrote:
On Fri, Nov 25, 2011 at 10:49:29AM +0000, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 06:42:42PM +0800, Osier Yang wrote:
On 2011年11月25日 18:28, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 04:55:02PM +0800, Osier Yang wrote:
<quote> AFAIU libvirt needs a way to:
- Associate a virtual adapter WWN with a VM (in the VM xml) - Learn to start a virtual adapter when the VM is started, and destroy the adapter when the VM is stopped. - Possibly a way to associate a WWN with a scsi pool, to start / stop a virtual adapter with the pool. </quote>
But afer thinking more, I'd think it might be not good idea:
As far as I could understand, the requirement of the BZ wants a way to create/migrate a guest with NPIV. I'm goint to talk create first,and migration then.
The desire to assocaite WWNN/WWPN with a VM is just one part of a more general need to expand libvirt's SCSI support. Paulo started a design thread on the subject a month or so back which sort of converged into agreement:
https://www.redhat.com/archives/libvir-list/2011-October/msg01253.html
Yes, I read this thread before, but it looks to me Paolo was talking about LUN, scsi host, and vHBA passthrough. It's a bit different with what I'm trying to resolve (There is no passthrough here, but about how to design a good workflow between virt-manager / Boxes and libvirt for using (creating and migration ) a FC LUN as a normal disk). The useful thing in the discussion of the thread may be define the (v)HBA as a controller though.
Or I misunderstood something?
I've found that when people generally talk about associating iSCSI LUNs with a guest, they have always been expecting SCSI LUN passthrough, or passthrough of the entire iSCSI vHBA.
Agreed, and IMO we should implement that also, but I think that work is divisible from the non-passthrough case, which is what we're considering.
If we're considering a non-passthrough use case, then IMHO the problem should be generalized to
How do we associated a storage volume with a VM ?
ie not something that is specific to (i)SCSI.
I like the idea of associating a storage volume with a particular block device on a VM.
So, what's your vision for how a VM would use storage that's visible to a virtual WWN? It sounds like you're saying that we should extend SCSI pools to take WWNN, WWPN and fabric name, and then instantiate the vHBA when the pool is started. We'd then extend the domain disk XML to take a pool/volume definition. When the VM is started, we'd check to see if the pool was active, if not, try to start it, and use the resulting volume as described in the domain XML. We'd have to manage the wait for the volumes to show up, but we have to contend with that regardless. That's kind of a nice option in this use case as it would allow administrators to bring the pool up ahead of domain start if desired.
Is that right?
That's not really what I had in mind for storage pools. I expect that the storage pool is configured by the mgmt app prior to creating the guest. When creating the guest they just refer to a volume within the pool. We shouldn't be auto-creating storage pools as a side effect of starting guests IMHO. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Nov 28, 2011 at 09:41:12PM +0000, Daniel P. Berrange wrote:
On Mon, Nov 28, 2011 at 09:01:45AM -0500, Dave Allan wrote:
On Fri, Nov 25, 2011 at 10:49:29AM +0000, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 06:42:42PM +0800, Osier Yang wrote:
On 2011年11月25日 18:28, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 04:55:02PM +0800, Osier Yang wrote:
<quote> AFAIU libvirt needs a way to:
- Associate a virtual adapter WWN with a VM (in the VM xml) - Learn to start a virtual adapter when the VM is started, and destroy the adapter when the VM is stopped. - Possibly a way to associate a WWN with a scsi pool, to start / stop a virtual adapter with the pool. </quote>
But afer thinking more, I'd think it might be not good idea:
As far as I could understand, the requirement of the BZ wants a way to create/migrate a guest with NPIV. I'm goint to talk create first,and migration then.
The desire to assocaite WWNN/WWPN with a VM is just one part of a more general need to expand libvirt's SCSI support. Paulo started a design thread on the subject a month or so back which sort of converged into agreement:
https://www.redhat.com/archives/libvir-list/2011-October/msg01253.html
Yes, I read this thread before, but it looks to me Paolo was talking about LUN, scsi host, and vHBA passthrough. It's a bit different with what I'm trying to resolve (There is no passthrough here, but about how to design a good workflow between virt-manager / Boxes and libvirt for using (creating and migration ) a FC LUN as a normal disk). The useful thing in the discussion of the thread may be define the (v)HBA as a controller though.
Or I misunderstood something?
I've found that when people generally talk about associating iSCSI LUNs with a guest, they have always been expecting SCSI LUN passthrough, or passthrough of the entire iSCSI vHBA.
Agreed, and IMO we should implement that also, but I think that work is divisible from the non-passthrough case, which is what we're considering.
If we're considering a non-passthrough use case, then IMHO the problem should be generalized to
How do we associated a storage volume with a VM ?
ie not something that is specific to (i)SCSI.
I like the idea of associating a storage volume with a particular block device on a VM.
So, what's your vision for how a VM would use storage that's visible to a virtual WWN? It sounds like you're saying that we should extend SCSI pools to take WWNN, WWPN and fabric name, and then instantiate the vHBA when the pool is started. We'd then extend the domain disk XML to take a pool/volume definition. When the VM is started, we'd check to see if the pool was active, if not, try to start it, and use the resulting volume as described in the domain XML. We'd have to manage the wait for the volumes to show up, but we have to contend with that regardless. That's kind of a nice option in this use case as it would allow administrators to bring the pool up ahead of domain start if desired.
Is that right?
That's not really what I had in mind for storage pools. I expect that the storage pool is configured by the mgmt app prior to creating the guest. When creating the guest they just refer to a volume within the pool. We shouldn't be auto-creating storage pools as a side effect of starting guests IMHO.
I think we're saying the same thing--I was thinking the user/mgmt app would create the pool, and starting the guest would start the pool if it wasn't already started, if a volume in it was associated with the VM. Dave
On 2011年11月29日 05:48, Dave Allan wrote:
On Mon, Nov 28, 2011 at 09:41:12PM +0000, Daniel P. Berrange wrote:
On Mon, Nov 28, 2011 at 09:01:45AM -0500, Dave Allan wrote:
On Fri, Nov 25, 2011 at 10:49:29AM +0000, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 06:42:42PM +0800, Osier Yang wrote:
On 2011年11月25日 18:28, Daniel P. Berrange wrote:
On Fri, Nov 25, 2011 at 04:55:02PM +0800, Osier Yang wrote:
> > <quote> > AFAIU libvirt needs a way to: > > - Associate a virtual adapter WWN with a VM (in the VM xml) > - Learn to start a virtual adapter when the VM is started, and destroy the > adapter when the VM is stopped. > - Possibly a way to associate a WWN with a scsi pool, to start / stop a > virtual adapter with the pool. > </quote> > > But afer thinking more, I'd think it might be not good idea: > > As far as I could understand, the requirement of the BZ wants > a way to create/migrate a guest with NPIV. I'm goint to talk > create first,and migration then.
The desire to assocaite WWNN/WWPN with a VM is just one part of a more general need to expand libvirt's SCSI support. Paulo started a design thread on the subject a month or so back which sort of converged into agreement:
https://www.redhat.com/archives/libvir-list/2011-October/msg01253.html
Yes, I read this thread before, but it looks to me Paolo was talking about LUN, scsi host, and vHBA passthrough. It's a bit different with what I'm trying to resolve (There is no passthrough here, but about how to design a good workflow between virt-manager / Boxes and libvirt for using (creating and migration ) a FC LUN as a normal disk). The useful thing in the discussion of the thread may be define the (v)HBA as a controller though.
Or I misunderstood something?
I've found that when people generally talk about associating iSCSI LUNs with a guest, they have always been expecting SCSI LUN passthrough, or passthrough of the entire iSCSI vHBA.
Agreed, and IMO we should implement that also, but I think that work is divisible from the non-passthrough case, which is what we're considering.
If we're considering a non-passthrough use case, then IMHO the problem should be generalized to
How do we associated a storage volume with a VM ?
ie not something that is specific to (i)SCSI.
I like the idea of associating a storage volume with a particular block device on a VM.
So, what's your vision for how a VM would use storage that's visible to a virtual WWN? It sounds like you're saying that we should extend SCSI pools to take WWNN, WWPN and fabric name, and then instantiate the vHBA when the pool is started. We'd then extend the domain disk XML to take a pool/volume definition. When the VM is started, we'd check to see if the pool was active, if not, try to start it, and use the resulting volume as described in the domain XML. We'd have to manage the wait for the volumes to show up, but we have to contend with that regardless. That's kind of a nice option in this use case as it would allow administrators to bring the pool up ahead of domain start if desired.
Is that right?
That's not really what I had in mind for storage pools. I expect that the storage pool is configured by the mgmt app prior to creating the guest. When creating the guest they just refer to a volume within the pool. We shouldn't be auto-creating storage pools as a side effect of starting guests IMHO.
I think we're saying the same thing--I was thinking the user/mgmt app would create the pool, and starting the guest would start the pool if it wasn't already started, if a volume in it was associated with the VM.
I'd agree with danpb more, i.e. it shouldn't mix the storage pool operation in domain's life. Both the storage and domain have public APIs for use, if we auto-create/start the storage pool as a side effect of domain starting, it probalbly means we need to call the storage's public API internally inside domain's public API, another way is to create some specific inernal storage functions for purpose, but it doesn't make things any better IMHO. It just could bring unexpected pain which we might suffer from in future, as the domain and storage are standalone modules with the original design, though I could't give an concrete proof now. Could anyone imagine it and give a proof? :-) Regards, Osier
On 11/28/2011 03:01 PM, Dave Allan wrote:
We'd then extend the domain disk XML to take a pool/volume definition.
I like the idea. It could be used for other things: - QEMU usermode iSCSI backend, including storing iSCSI secrets outside the domain XML. In this case, multiple domains will usually share the pool and specify different volumes. - SCSI controller passthrough: instead of defining the destination inside the domain XML, just refer to a SCSI (or perhaps iSCSI) pool. NPIV ends up being a special case. However, for the case of controller passthrough, wouldn't you end up basically with a separate pool for each domain? Especially for NPIV where each domain will use a separate WWPN/WWNN? It's still an advantage compared to duplicating the WWPN/WWNN once per LUN, but it may be a bit complex to manage. Paolo
participants (4)
-
Daniel P. Berrange -
Dave Allan -
Osier Yang -
Paolo Bonzini