[libvirt] VN-Link vNIC memory state copying on VM Migration
Hi All, I am new to libvirt. And want to know how the VM migration occurs in VN-Link (IEEE802.1Qbh). As far as I know, the memory state of vNICs in M81KR VIC has to be copied and moved to the destination vNIC on VM migration. Is that correct? If so, could you please tell me how this has been implemented in libvirt? Thanks Jagath
Jagath, what memory state do you refer to exactly? If you refer to port profile info, then there is no copying involved: the source host disassociates the vnic port profile and the destination host re-associates the port profile on the new vnic. /Chris
-----Original Message----- From: libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com] On Behalf Of Jagath Weerasinghe Sent: Friday, April 13, 2012 9:16 AM To: libvir-list@redhat.com Subject: [libvirt] VN-Link vNIC memory state copying on VM Migration
Hi All,
I am new to libvirt. And want to know how the VM migration occurs in VN-Link (IEEE802.1Qbh). As far as I know, the memory state of vNICs in M81KR VIC has to be copied and moved to the destination vNIC on VM migration. Is that correct? If so, could you please tell me how this has been implemented in libvirt?
Thanks Jagath
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
what memory state do you refer to exactly? What I meant by the memory state is the traffic being
Chris, Thanks for the reply. transferred through the vNIC when VM migration is triggered. If this memory state (or traffic) is not copied and moved to the destination vNIC how the smooth communication, which is guaranteed on VM migration, is achieved? Thanks Jagath
If you refer to port profile info, then there is no copying involved: the source host disassociates the vnic port profile and the destination host re-associates the port profile on the new vnic.
/Chris
-----Original Message----- From: libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com] On Behalf Of Jagath Weerasinghe Sent: Friday, April 13, 2012 9:16 AM To: libvir-list@redhat.com Subject: [libvirt] VN-Link vNIC memory state copying on VM Migration
Hi All,
I am new to libvirt. And want to know how the VM migration occurs in VN-Link (IEEE802.1Qbh). As far as I know, the memory state of vNICs in M81KR VIC has to be copied and moved to the destination vNIC on VM migration. Is that correct? If so, could you please tell me how this has been implemented in libvirt?
Thanks Jagath
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Jagath,
-----Original Message----- From: libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com] On Behalf Of Jagath Weerasinghe Sent: Friday, April 13, 2012 6:17 PM To: libvir-list@redhat.com Subject: Re: [libvirt] VN-Link vNIC memory state copying on VM Migration
Chris,
Thanks for the reply.
what memory state do you refer to exactly? What I meant by the memory state is the traffic being transferred through the vNIC when VM migration is triggered.
Since you are worried about the traffic I assume we are talking about _live_ migration. Libvirt qemu migration V3 takes place in 5 phases. (If you search the mailing list archive for "migration v3" you will find a number of 3Ds) This is a simplified description of the algorithm: +--+ |VM| +--------+ +---------+ |Src host| |dst host | +--------+ +---------+ Phase1: BEGIN - Get XML of VM and tx it to dst Phase2: PREPARE - create (paused) VM based on XML Phase3: PERFORM - Copy RAM to dst - Pause VM .................................................. Phase4: FINISH - Apply port profiles .................................................. - Start VM Phase5: CONFIRM - Kill VM Libvirt on the dst host orchestrates everything using RPCs. Dst host can return error/success codes (the two libvirt also exchange data, .. see cookies) During phases 1/2/3 the VM is still running on the src host and hence able to rx/tx traffic. At the end of Phase 3, when qemu is done copying the RAM from source host to destination host (*), qemu puts the VM in pause state on the source host. (*) See the migration parameter 'migrate-setmaxdowntime' In phase 4 libvirt applies the port profile to the interfaces that need it and starts the VM (which had been created and put in pause state in phase 2). There is therefore a small period during which both VMs (the one on the source host and the one on the destination host) are in pause state (see dotted lines above). Phase 4 will start the VM on the dst host, Phase 5 will kill the (paused) VM on the src host. The amount of time during which both VMs are in pause state depends on a number of factors, including: - maxdowntime: the smaller you configure this value and - the longer will take the migration to complete, but - the smaller the pause duration will be - bandwidth available on the interface/s used to carry the migration traffic, and amount of RAM assigned to the VM (since you need to copy it) - time taken to complete the port profile associations on the dst host (which depends on how loaded the switches are) - ... Hope this clarifies a bit. /Chris
If this memory state (or traffic) is not copied and moved to the destination vNIC how the smooth communication, which is guaranteed on VM migration, is achieved?
Thanks Jagath
If you refer to port profile info, then there is no copying involved: the source host disassociates the vnic port profile and the destination host re-associates the port profile on the new vnic.
/Chris
-----Original Message----- From: libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com] On Behalf Of Jagath Weerasinghe Sent: Friday, April 13, 2012 9:16 AM To: libvir-list@redhat.com Subject: [libvirt] VN-Link vNIC memory state copying on VM Migration
Hi All,
I am new to libvirt. And want to know how the VM migration occurs in VN-Link (IEEE802.1Qbh). As far as I know, the memory state of vNICs in M81KR VIC has to be copied and moved to the destination vNIC on VM migration. Is that correct? If so, could you please tell me how this has been implemented in libvirt?
Thanks Jagath
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Chris, Thanks the explanation of the algorithm.
what memory state do you refer to exactly? What I meant by the memory state is the traffic being transferred through the vNIC when VM migration is triggered.
Since you are worried about the traffic I assume we are talking about _live_ migration.
Yes, live migration. Sorry for not mentioning it clearly.
Libvirt qemu migration V3 takes place in 5 phases. (If you search the mailing list archive for "migration v3" you will find a number of 3Ds) This is a simplified description of the algorithm:
+--+ |VM| +--------+ +---------+ |Src host| |dst host | +--------+ +---------+
Phase1: BEGIN - Get XML of VM and tx it to dst Phase2: PREPARE - create (paused) VM based on XML Phase3: PERFORM - Copy RAM to dst - Pause VM .................................................. Phase4: FINISH - Apply port profiles .................................................. - Start VM Phase5: CONFIRM - Kill VM
Libvirt on the dst host orchestrates everything using RPCs. Dst host can return error/success codes (the two libvirt also exchange data, .. see cookies)
During phases 1/2/3 the VM is still running on the src host and hence able to rx/tx traffic. At the end of Phase 3, when qemu is done copying the RAM from source host to destination host (*), qemu puts the VM in pause state on the source host.
OK. In phase 3, the RAM used by VM in source host is copied to the destination. Then what about the memory state of vNIC's in the M81KR VIC. Isn't there any memory for traffic passing through vNICs and also for vNIC statistics? What I am talking about is copying of this memory to the destination vNIC. Thanks Jagath
(*) See the migration parameter 'migrate-setmaxdowntime'
In phase 4 libvirt applies the port profile to the interfaces that need it and starts the VM (which had been created and put in pause state in phase 2). There is therefore a small period during which both VMs (the one on the source host and the one on the destination host) are in pause state (see dotted lines above). Phase 4 will start the VM on the dst host, Phase 5 will kill the (paused) VM on the src host.
The amount of time during which both VMs are in pause state depends on a number of factors, including: - maxdowntime: the smaller you configure this value and - the longer will take the migration to complete, but - the smaller the pause duration will be - bandwidth available on the interface/s used to carry the migration traffic, and amount of RAM assigned to the VM (since you need to copy it) - time taken to complete the port profile associations on the dst host (which depends on how loaded the switches are) - ...
Hope this clarifies a bit. /Chris
If this memory state (or traffic) is not copied and moved to the destination vNIC how the smooth communication, which is guaranteed on VM migration, is achieved?
Thanks Jagath
If you refer to port profile info, then there is no copying involved: the source host disassociates the vnic port profile and the destination host re-associates the port profile on the new vnic.
/Chris
-----Original Message----- From: libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com] On Behalf Of Jagath Weerasinghe Sent: Friday, April 13, 2012 9:16 AM To: libvir-list@redhat.com Subject: [libvirt] VN-Link vNIC memory state copying on VM Migration
Hi All,
I am new to libvirt. And want to know how the VM migration occurs in VN-Link (IEEE802.1Qbh). As far as I know, the memory state of vNICs in M81KR VIC has to be copied and moved to the destination vNIC on VM migration. Is that correct? If so, could you please tell me how this has been implemented in libvirt?
Thanks Jagath
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Jagath,
-----Original Message----- From: Jagath Weerasinghe [mailto:jagfoss@gmail.com] Sent: Sunday, April 15, 2012 6:59 AM To: Christian Benvenuti (benve) Cc: libvir-list@redhat.com Subject: Re: [libvirt] VN-Link vNIC memory state copying on VM Migration
Chris,
Thanks the explanation of the algorithm.
what memory state do you refer to exactly? What I meant by the memory state is the traffic being transferred through the vNIC when VM migration is triggered.
Since you are worried about the traffic I assume we are talking about _live_ migration.
Yes, live migration. Sorry for not mentioning it clearly.
Libvirt qemu migration V3 takes place in 5 phases. (If you search the mailing list archive for "migration v3" you will find a number of 3Ds) This is a simplified description of the algorithm:
+--+ |VM| +--------+ +---------+ |Src host| |dst host | +--------+ +---------+
Phase1: BEGIN - Get XML of VM and tx it to dst Phase2: PREPARE - create (paused) VM based on XML Phase3: PERFORM - Copy RAM to dst - Pause VM .................................................. Phase4: FINISH - Apply port profiles .................................................. - Start VM Phase5: CONFIRM - Kill VM
Libvirt on the dst host orchestrates everything using RPCs. Dst host can return error/success codes (the two libvirt also exchange data, .. see cookies)
During phases 1/2/3 the VM is still running on the src host and hence able to rx/tx traffic. At the end of Phase 3, when qemu is done copying the RAM from source host to destination host (*), qemu puts the VM in pause state on the source host.
OK. In phase 3, the RAM used by VM in source host is copied to the destination. Then what about the memory state of vNIC's in the M81KR VIC. Isn't there any memory for traffic passing through vNICs and also for vNIC statistics? What I am talking about is copying of this memory to the destination vNIC.
No, that data (statistics, etc) are not copied to the dst host/vnic. However, you can find those statistics/counters on the switch (which are preserved across live migrations). Thanks /Chris
(*) See the migration parameter 'migrate-setmaxdowntime'
In phase 4 libvirt applies the port profile to the interfaces that need it and starts the VM (which had been created and put in pause state in phase 2). There is therefore a small period during which both VMs (the one on the source host and the one on the destination host) are in pause state (see dotted lines above). Phase 4 will start the VM on the dst host, Phase 5 will kill the (paused) VM on the src host.
The amount of time during which both VMs are in pause state depends on a number of factors, including: - maxdowntime: the smaller you configure this value and - the longer will take the migration to complete, but - the smaller the pause duration will be - bandwidth available on the interface/s used to carry the migration traffic, and amount of RAM assigned to the VM (since you need to copy it) - time taken to complete the port profile associations on the dst host (which depends on how loaded the switches are) - ...
Hope this clarifies a bit. /Chris
If this memory state (or traffic) is not copied and moved to the destination vNIC how the smooth communication, which is guaranteed on VM migration, is achieved?
Thanks Jagath
If you refer to port profile info, then there is no copying involved: the source host disassociates the vnic port profile and the destination host re-associates the port profile on the new vnic.
/Chris
-----Original Message----- From: libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com] On Behalf Of Jagath Weerasinghe Sent: Friday, April 13, 2012 9:16 AM To: libvir-list@redhat.com Subject: [libvirt] VN-Link vNIC memory state copying on VM Migration
Hi All,
I am new to libvirt. And want to know how the VM migration occurs in VN-Link (IEEE802.1Qbh). As far as I know, the memory state of vNICs in M81KR VIC has to be copied and moved to the destination vNIC on VM migration. Is that correct? If so, could you please tell me how this has been implemented in libvirt?
Thanks Jagath
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Chris
OK. In phase 3, the RAM used by VM in source host is copied to the destination. Then what about the memory state of vNIC's in the M81KR VIC. Isn't there any memory for traffic passing through vNICs and also for vNIC statistics? What I am talking about is copying of this memory to the destination vNIC.
No, that data (statistics, etc) are not copied to the dst host/vnic. However, you can find those statistics/counters on the switch (which are preserved across live migrations).
OK. I think this operation is the same for both IEEE802.1qbh (VN-Link) and IEEE802.1qbg. Is that correct? One more thing. Could you please tell me where the port profile association/dissociation messages of libvirt are converted to VIC protocol messages. Is this done by CNA it self? Thanks Jagath
Thanks /Chris
(*) See the migration parameter 'migrate-setmaxdowntime'
In phase 4 libvirt applies the port profile to the interfaces that need it and starts the VM (which had been created and put in pause state in phase 2). There is therefore a small period during which both VMs (the one on the source host and the one on the destination host) are in pause state (see dotted lines above). Phase 4 will start the VM on the dst host, Phase 5 will kill the (paused) VM on the src host.
The amount of time during which both VMs are in pause state depends on a number of factors, including: - maxdowntime: the smaller you configure this value and - the longer will take the migration to complete, but - the smaller the pause duration will be - bandwidth available on the interface/s used to carry the migration traffic, and amount of RAM assigned to the VM (since you need to copy it) - time taken to complete the port profile associations on the dst host (which depends on how loaded the switches are) - ...
Hope this clarifies a bit. /Chris
If this memory state (or traffic) is not copied and moved to the destination vNIC how the smooth communication, which is guaranteed on VM migration, is achieved?
Thanks Jagath
If you refer to port profile info, then there is no copying involved: the source host disassociates the vnic port profile and the destination host re-associates the port profile on the new vnic.
/Chris
-----Original Message----- From: libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com] On Behalf Of Jagath Weerasinghe Sent: Friday, April 13, 2012 9:16 AM To: libvir-list@redhat.com Subject: [libvirt] VN-Link vNIC memory state copying on VM Migration
Hi All,
I am new to libvirt. And want to know how the VM migration occurs in VN-Link (IEEE802.1Qbh). As far as I know, the memory state of vNICs in M81KR VIC has to be copied and moved to the destination vNIC on VM migration. Is that correct? If so, could you please tell me how this has been implemented in libvirt?
Thanks Jagath
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-----Original Message----- From: Jagath Weerasinghe [mailto:jagfoss@gmail.com] Sent: Tuesday, April 17, 2012 8:19 AM To: Christian Benvenuti (benve) Cc: libvir-list@redhat.com Subject: Re: [libvirt] VN-Link vNIC memory state copying on VM Migration
Chris
OK. In phase 3, the RAM used by VM in source host is copied to the destination. Then what about the memory state of vNIC's in the M81KR VIC. Isn't there any memory for traffic passing through vNICs and also for vNIC statistics? What I am talking about is copying of this memory to the destination vNIC.
No, that data (statistics, etc) are not copied to the dst host/vnic. However, you can find those statistics/counters on the switch (which are preserved across live migrations).
OK. I think this operation is the same for both IEEE802.1qbh (VN-Link) and IEEE802.1qbg. Is that correct?
I assume so.
One more thing. Could you please tell me where the port profile association/dissociation messages of libvirt are converted to VIC protocol messages. Is this done by CNA it self?
Yes, the NIC firmware takes care of it. /Chris
(*) See the migration parameter 'migrate-setmaxdowntime'
In phase 4 libvirt applies the port profile to the interfaces that need it and starts the VM (which had been created and put in pause state in phase 2). There is therefore a small period during which both VMs (the one on the source host and the one on the destination host) are in pause state (see dotted lines above). Phase 4 will start the VM on the dst host, Phase 5 will kill the (paused) VM on the src host.
The amount of time during which both VMs are in pause state depends on a number of factors, including: - maxdowntime: the smaller you configure this value and - the longer will take the migration to complete, but - the smaller the pause duration will be - bandwidth available on the interface/s used to carry the migration traffic, and amount of RAM assigned to the VM (since you need to copy it) - time taken to complete the port profile associations on the dst host (which depends on how loaded the switches are) - ...
Hope this clarifies a bit. /Chris
If this memory state (or traffic) is not copied and moved to the destination vNIC how the smooth communication, which is guaranteed on VM migration, is achieved?
Thanks Jagath
If you refer to port profile info, then there is no copying involved: the source host disassociates the vnic port profile and the destination host re-associates the port profile on the new vnic.
/Chris
> -----Original Message----- > From: libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com] On Behalf Of Jagath > Weerasinghe > Sent: Friday, April 13, 2012 9:16 AM > To: libvir-list@redhat.com > Subject: [libvirt] VN-Link vNIC memory state copying on VM Migration > > Hi All, > > I am new to libvirt. And want to know how the VM migration > occurs in VN-Link (IEEE802.1Qbh). As far as I know, > the memory state of vNICs in M81KR VIC has to be copied > and moved to the destination vNIC on VM migration. > Is that correct? If so, could you please tell me how this > has been implemented in libvirt? > > Thanks > Jagath > > -- > libvir-list mailing list > libvir-list@redhat.com > https://www.redhat.com/mailman/listinfo/libvir-list
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Chris Thanks Jagath On Wed, Apr 18, 2012 at 3:40 AM, Christian Benvenuti (benve) <benve@cisco.com> wrote:
-----Original Message----- From: Jagath Weerasinghe [mailto:jagfoss@gmail.com] Sent: Tuesday, April 17, 2012 8:19 AM To: Christian Benvenuti (benve) Cc: libvir-list@redhat.com Subject: Re: [libvirt] VN-Link vNIC memory state copying on VM Migration
Chris
OK. In phase 3, the RAM used by VM in source host is copied to the destination. Then what about the memory state of vNIC's in the M81KR VIC. Isn't there any memory for traffic passing through vNICs and also for vNIC statistics? What I am talking about is copying of this memory to the destination vNIC.
No, that data (statistics, etc) are not copied to the dst host/vnic. However, you can find those statistics/counters on the switch (which are preserved across live migrations).
OK. I think this operation is the same for both IEEE802.1qbh (VN-Link) and IEEE802.1qbg. Is that correct?
I assume so.
One more thing. Could you please tell me where the port profile association/dissociation messages of libvirt are converted to VIC protocol messages. Is this done by CNA it self?
Yes, the NIC firmware takes care of it.
/Chris
(*) See the migration parameter 'migrate-setmaxdowntime'
In phase 4 libvirt applies the port profile to the interfaces that need it and starts the VM (which had been created and put in pause state in phase 2). There is therefore a small period during which both VMs (the one on the source host and the one on the destination host) are in pause state (see dotted lines above). Phase 4 will start the VM on the dst host, Phase 5 will kill the (paused) VM on the src host.
The amount of time during which both VMs are in pause state depends on a number of factors, including: - maxdowntime: the smaller you configure this value and - the longer will take the migration to complete, but - the smaller the pause duration will be - bandwidth available on the interface/s used to carry the migration traffic, and amount of RAM assigned to the VM (since you need to copy it) - time taken to complete the port profile associations on the dst host (which depends on how loaded the switches are) - ...
Hope this clarifies a bit. /Chris
If this memory state (or traffic) is not copied and moved to the destination vNIC how the smooth communication, which is guaranteed on VM migration, is achieved?
Thanks Jagath
> > If you refer to port profile info, then there is no copying involved: > the source host disassociates the vnic port profile and > the destination host re-associates the port profile on the new vnic. > > /Chris > >> -----Original Message----- >> From: libvir-list-bounces@redhat.com > [mailto:libvir-list-bounces@redhat.com] On Behalf Of Jagath >> Weerasinghe >> Sent: Friday, April 13, 2012 9:16 AM >> To: libvir-list@redhat.com >> Subject: [libvirt] VN-Link vNIC memory state copying on VM Migration >> >> Hi All, >> >> I am new to libvirt. And want to know how the VM migration >> occurs in VN-Link (IEEE802.1Qbh). As far as I know, >> the memory state of vNICs in M81KR VIC has to be copied >> and moved to the destination vNIC on VM migration. >> Is that correct? If so, could you please tell me how this >> has been implemented in libvirt? >> >> Thanks >> Jagath >> >> -- >> libvir-list mailing list >> libvir-list@redhat.com >> https://www.redhat.com/mailman/listinfo/libvir-list
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
participants (2)
-
Christian Benvenuti (benve) -
Jagath Weerasinghe