[libvirt] sending openvswitch "port db" info during libvirt migration
Someone asked on IRC the other day about sending openvswitch per-port data (normally stored in the switch) to the destination host during a migration. I suggested maybe this could be handled by encoding the information into the interface's <virtualport> prior to migration, and then writing it back out to openvswitch on the destination when the interface was reconnected there. I think there's a problem with that, though - they don't want to save the port data on the source until the instant that the interface is disconnected (since it is constantly changing), but by then the domain XML has already long ago been formatted and sent to the destination. So is there some other way within the confines of the current migration protocol that this information can be sent from migration source to destination?
On Tue, Aug 07, 2012 at 03:50:20PM -0400, Laine Stump wrote:
Someone asked on IRC the other day about sending openvswitch per-port data (normally stored in the switch) to the destination host during a migration. I suggested maybe this could be handled by encoding the information into the interface's <virtualport> prior to migration, and then writing it back out to openvswitch on the destination when the interface was reconnected there.
I think there's a problem with that, though - they don't want to save the port data on the source until the instant that the interface is disconnected (since it is constantly changing), but by then the domain XML has already long ago been formatted and sent to the destination.
So is there some other way within the confines of the current migration protocol that this information can be sent from migration source to destination?
I'd be interested to know more about just what sort of data it is that needs to be passed around. From what you describe though, the only way to pass that info would be to store it in the migration cookie when the 'Perform' step completes (ie QEMU now paused, no network I/O taking place), whereupon it will be passed into the 'Finish' step on the dest (which can set it and then resume QEMU) Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Aug 8, 2012, at 9:49 AM, Daniel P. Berrange wrote:
On Tue, Aug 07, 2012 at 03:50:20PM -0400, Laine Stump wrote:
Someone asked on IRC the other day about sending openvswitch per-port data (normally stored in the switch) to the destination host during a migration. I suggested maybe this could be handled by encoding the information into the interface's <virtualport> prior to migration, and then writing it back out to openvswitch on the destination when the interface was reconnected there.
I think there's a problem with that, though - they don't want to save the port data on the source until the instant that the interface is disconnected (since it is constantly changing), but by then the domain XML has already long ago been formatted and sent to the destination.
So is there some other way within the confines of the current migration protocol that this information can be sent from migration source to destination?
I'd be interested to know more about just what sort of data it is that needs to be passed around. From what you describe though, the only way to pass that info would be to store it in the migration cookie when the 'Perform' step completes (ie QEMU now paused, no network I/O taking place), whereupon it will be passed into the 'Finish' step on the dest (which can set it and then resume QEMU)
This is per-port data, tied to the specific virtual port associated with the VM itself. This data is stored in the OVS DB, and is used/could be used by an entity controlling Open vSwitch on the host. For example, stats may be kept here.
From what you indicate above, can you point me at the code which handles the migration cookie?
Thanks! Kyle
Daniel
On 08/08/2012 12:30 PM, Kyle Mestery (kmestery) wrote:
On Aug 8, 2012, at 9:49 AM, Daniel P. Berrange wrote:
On Tue, Aug 07, 2012 at 03:50:20PM -0400, Laine Stump wrote:
Someone asked on IRC the other day about sending openvswitch per-port data (normally stored in the switch) to the destination host during a migration. I suggested maybe this could be handled by encoding the information into the interface's <virtualport> prior to migration, and then writing it back out to openvswitch on the destination when the interface was reconnected there.
I think there's a problem with that, though - they don't want to save the port data on the source until the instant that the interface is disconnected (since it is constantly changing), but by then the domain XML has already long ago been formatted and sent to the destination.
So is there some other way within the confines of the current migration protocol that this information can be sent from migration source to destination? I'd be interested to know more about just what sort of data it is that needs to be passed around. From what you describe though, the only way to pass that info would be to store it in the migration cookie when the 'Perform' step completes (ie QEMU now paused, no network I/O taking place), whereupon it will be passed into the 'Finish' step on the dest (which can set it and then resume QEMU)
This is per-port data, tied to the specific virtual port associated with the VM itself. This data is stored in the OVS DB, and is used/could be used by an entity controlling Open vSwitch on the host. For example, stats may be kept here.
From what you indicate above, can you point me at the code which handles the migration cookie?
Look in qemu_migration.c. I notice there is also code there that calls the 802.1QbX-oriented virtualport associate function (search for qemuMigrationVPAssociatePortProfiles), and it's likely there where the port data should be unloaded from the cookie into the destination host. As for when/where to load the port data *into* the cookie on the source, I'm new to that code too, and didn't see it right away :-) I do see that the entire persistent definition can be sent in the cookie. It seems to me that "last minute (last instant) updates" to the live XML could be an issue not just for interfaces connected to ovs switches, but for other circumstances as well. Does it maybe make sense to solve this in a more generalized fashion, by resending the complete live XML? (I'm assuming that during migration libvirt is preventing any modification to the live XML that would result in invalidating the vm state contained in the memory image being transferred).
On Wed, Aug 08, 2012 at 10:58:28PM -0400, Laine Stump wrote:
On 08/08/2012 12:30 PM, Kyle Mestery (kmestery) wrote:
On Aug 8, 2012, at 9:49 AM, Daniel P. Berrange wrote:
On Tue, Aug 07, 2012 at 03:50:20PM -0400, Laine Stump wrote:
Someone asked on IRC the other day about sending openvswitch per-port data (normally stored in the switch) to the destination host during a migration. I suggested maybe this could be handled by encoding the information into the interface's <virtualport> prior to migration, and then writing it back out to openvswitch on the destination when the interface was reconnected there.
I think there's a problem with that, though - they don't want to save the port data on the source until the instant that the interface is disconnected (since it is constantly changing), but by then the domain XML has already long ago been formatted and sent to the destination.
So is there some other way within the confines of the current migration protocol that this information can be sent from migration source to destination? I'd be interested to know more about just what sort of data it is that needs to be passed around. From what you describe though, the only way to pass that info would be to store it in the migration cookie when the 'Perform' step completes (ie QEMU now paused, no network I/O taking place), whereupon it will be passed into the 'Finish' step on the dest (which can set it and then resume QEMU)
This is per-port data, tied to the specific virtual port associated with the VM itself. This data is stored in the OVS DB, and is used/could be used by an entity controlling Open vSwitch on the host. For example, stats may be kept here.
From what you indicate above, can you point me at the code which handles the migration cookie?
Look in qemu_migration.c. I notice there is also code there that calls the 802.1QbX-oriented virtualport associate function (search for qemuMigrationVPAssociatePortProfiles), and it's likely there where the port data should be unloaded from the cookie into the destination host. As for when/where to load the port data *into* the cookie on the source, I'm new to that code too, and didn't see it right away :-)
I do see that the entire persistent definition can be sent in the cookie. It seems to me that "last minute (last instant) updates" to the live XML could be an issue not just for interfaces connected to ovs switches, but for other circumstances as well. Does it maybe make sense to solve this in a more generalized fashion, by resending the complete live XML? (I'm assuming that during migration libvirt is preventing any modification to the live XML that would result in invalidating the vm state contained in the memory image being transferred).
No, at that point the VM is already started on the dest host, so you can't just re-send the live XML there. You can put any other data into the cookie though, it doesn't have to be passed in the persistent XML directly. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Aug 9, 2012, at 4:23 AM, Daniel P. Berrange wrote:
On Wed, Aug 08, 2012 at 10:58:28PM -0400, Laine Stump wrote:
On 08/08/2012 12:30 PM, Kyle Mestery (kmestery) wrote:
On Aug 8, 2012, at 9:49 AM, Daniel P. Berrange wrote:
On Tue, Aug 07, 2012 at 03:50:20PM -0400, Laine Stump wrote:
Someone asked on IRC the other day about sending openvswitch per-port data (normally stored in the switch) to the destination host during a migration. I suggested maybe this could be handled by encoding the information into the interface's <virtualport> prior to migration, and then writing it back out to openvswitch on the destination when the interface was reconnected there.
I think there's a problem with that, though - they don't want to save the port data on the source until the instant that the interface is disconnected (since it is constantly changing), but by then the domain XML has already long ago been formatted and sent to the destination.
So is there some other way within the confines of the current migration protocol that this information can be sent from migration source to destination? I'd be interested to know more about just what sort of data it is that needs to be passed around. From what you describe though, the only way to pass that info would be to store it in the migration cookie when the 'Perform' step completes (ie QEMU now paused, no network I/O taking place), whereupon it will be passed into the 'Finish' step on the dest (which can set it and then resume QEMU)
This is per-port data, tied to the specific virtual port associated with the VM itself. This data is stored in the OVS DB, and is used/could be used by an entity controlling Open vSwitch on the host. For example, stats may be kept here.
From what you indicate above, can you point me at the code which handles the migration cookie?
Look in qemu_migration.c. I notice there is also code there that calls the 802.1QbX-oriented virtualport associate function (search for qemuMigrationVPAssociatePortProfiles), and it's likely there where the port data should be unloaded from the cookie into the destination host. As for when/where to load the port data *into* the cookie on the source, I'm new to that code too, and didn't see it right away :-)
I do see that the entire persistent definition can be sent in the cookie. It seems to me that "last minute (last instant) updates" to the live XML could be an issue not just for interfaces connected to ovs switches, but for other circumstances as well. Does it maybe make sense to solve this in a more generalized fashion, by resending the complete live XML? (I'm assuming that during migration libvirt is preventing any modification to the live XML that would result in invalidating the vm state contained in the memory image being transferred).
No, at that point the VM is already started on the dest host, so you can't just re-send the live XML there. You can put any other data into the cookie though, it doesn't have to be passed in the persistent XML directly.
Daniel
Thanks guys, I think I see how to do this now. I'll work on this and send some patches. Thanks, Kyle
participants (3)
-
Daniel P. Berrange -
Kyle Mestery (kmestery) -
Laine Stump