[libvirt-users] vm live storage migration with snapshots
Hi all, I'm investigating the ways to improve the live migration performance in libvirt. I have a question about the vm live storage migration. the platform is libvirt + qemu + kvm If we want to migrate a running vm with its virtual disk images to another node. we can use 'virsh migrate ....' commands. What if this vm has a number of disk-only external snapshots? In the current version, how can live migrate this vm? Is it possible to blockcommit the snapshots to the base image and later perform the migration, without shutting down the running vm? Or is it possible to iteratively transfer all the snapshots to the destination and later live migrate only the latest new data? Thanks in advance. Best, Ed
On 02/11/2015 10:08 AM, Edward Young wrote:
Hi all,
[probably didn't need to cross-post to quite that wide of an audience, oh well]
I'm investigating the ways to improve the live migration performance in libvirt. I have a question about the vm live storage migration. the platform is libvirt + qemu + kvm
If we want to migrate a running vm with its virtual disk images to another node. we can use 'virsh migrate ....' commands.
What if this vm has a number of disk-only external snapshots? In the current version, how can live migrate this vm?
Are the snapshots based on shared storage, or local-only storage? If I understand your question correctly, you are starting with this situation: base <- mid <- active and want to make sure that on the destination, the guest still sees the same contents as 'active' on the source, whether or not the destination still sees a backing chain or just one file.
Is it possible to blockcommit the snapshots to the base image and later perform the migration, without shutting down the running vm?
Yes, but that may not be the fastest solution. That is, you would be going from: base <- mid <- active to base' where base' is the active image containing all the committed state, then migrate that file. If base is not shared, that means qemu has to migrate the entire disk state. And you also no longer have the external snapshots to revert to on the destination.
Or is it possible to iteratively transfer all the snapshots to the destination and later live migrate only the latest new data?
Yes, that works too, and is probably faster, especially if you have out-of-band means for sharing read-only state between source and destination. In fact, if you can create yet another (temporary) external snapshot, but this time on shared storage, and have storage backends that let you instantly remap backing files to be visible into the destination, then it is as simple as: create a snapshot with shared storage destination base <- mid <- active(frozen) <- shared copy base, mid, and active to the destination live migrate using shared storage instead of doing storage migration block-commit shared back into active base <- mid <- active where the shared storage only needs to hold as much disk state as diverges during the time of the live migration. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Hi Eric, Thanks for your reply! I have the follow up questions blew. On Wed, Feb 11, 2015 at 11:52 AM, Eric Blake <eblake@redhat.com> wrote:
On 02/11/2015 10:08 AM, Edward Young wrote:
Hi all,
[probably didn't need to cross-post to quite that wide of an audience, oh well]
I'm investigating the ways to improve the live migration performance in libvirt. I have a question about the vm live storage migration. the platform is libvirt + qemu + kvm
If we want to migrate a running vm with its virtual disk images to
another
node. we can use 'virsh migrate ....' commands.
What if this vm has a number of disk-only external snapshots? In the current version, how can live migrate this vm?
Are the snapshots based on shared storage, or local-only storage?
Yes, I'm talking about the local-only storage.
If I understand your question correctly, you are starting with this situation:
base <- mid <- active
and want to make sure that on the destination, the guest still sees the same contents as 'active' on the source, whether or not the destination still sees a backing chain or just one file.
Is it possible to blockcommit the snapshots to the base image and later perform the migration, without shutting down the running vm?
Yes, but that may not be the fastest solution. That is, you would be going from:
base <- mid <- active
to
base'
where base' is the active image containing all the committed state, then migrate that file. If base is not shared, that means qemu has to migrate the entire disk state. And you also no longer have the external snapshots to revert to on the destination.
Yes, I agree with you. In this case, we need to migrate the entire disk state. In this case, there is no snapshot involved. we just perform the regular migration with 'virsh migrate....'. Is this correct?
Or is it possible to iteratively transfer all the snapshots to the destination and later live migrate only the latest new data?
Yes, that works too, and is probably faster, especially if you have out-of-band means for sharing read-only state between source and destination.
My question is here. If we do not have any shared storage resource between source and destination (eg. long distance VM migration), how can we migrate the latest new data to the destination? we can copy the base, mid to destination manually, then how can we migrate the active snapshot( new data goes in)? I learned that drive_mirror in qemu is built to finish this, but do not understand clearly. Could you elaborate for me, or provide an example?
In fact, if you can create yet another (temporary) external snapshot, but this time on shared storage, and have storage backends that let you instantly remap backing files to be visible into the destination, then it is as simple as:
create a snapshot with shared storage destination base <- mid <- active(frozen) <- shared copy base, mid, and active to the destination live migrate using shared storage instead of doing storage migration block-commit shared back into active base <- mid <- active
where the shared storage only needs to hold as much disk state as diverges during the time of the live migration.
Thanks a lot! Ed
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 02/11/2015 02:07 PM, Edward Young wrote:
What if this vm has a number of disk-only external snapshots? In the current version, how can live migrate this vm?
Are the snapshots based on shared storage, or local-only storage?
Yes, I'm talking about the local-only storage.
Okay, glad I guessed as much, then.
Yes, I agree with you. In this case, we need to migrate the entire disk state. In this case, there is no snapshot involved. we just perform the regular migration with 'virsh migrate....'. Is this correct?
Using 'virsh migrate --copy-storage-all' would indeed migrate the entire disk, if you can't supply shared storage.
Or is it possible to iteratively transfer all the snapshots to the destination and later live migrate only the latest new data?
Yes, that works too, and is probably faster, especially if you have out-of-band means for sharing read-only state between source and destination.
My question is here. If we do not have any shared storage resource between source and destination (eg. long distance VM migration), how can we migrate the latest new data to the destination? we can copy the base, mid to destination manually, then how can we migrate the active snapshot( new data goes in)? I learned that drive_mirror in qemu is built to finish this, but do not understand clearly. Could you elaborate for me, or provide an example?
Use 'virsh migrate --copy-storage-inc' to migrate only the incremental changes, which assumes that the destination can already see the same read-only backing data that the source sees. In fact, modern libvirt/qemu does this for you by setting up an NBD server on the destination, doing a data mirror from the source into the destination (so that you DO have shared storage, at least for the duration of the migration), then doing the live migration, then tearing down the NBD mirroring link. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Hi Eric, Please see the blew: On Wed, Feb 11, 2015 at 3:12 PM, Eric Blake <eblake@redhat.com> wrote:
On 02/11/2015 02:07 PM, Edward Young wrote:
What if this vm has a number of disk-only external snapshots? In the current version, how can live migrate this vm?
Are the snapshots based on shared storage, or local-only storage?
Yes, I'm talking about the local-only storage.
Okay, glad I guessed as much, then.
Yes, I agree with you. In this case, we need to migrate the entire disk state. In this case, there is no snapshot involved. we just perform the regular migration with 'virsh migrate....'. Is this correct?
Using 'virsh migrate --copy-storage-all' would indeed migrate the entire disk, if you can't supply shared storage.
Or is it possible to iteratively transfer all the snapshots to the destination and later live migrate only the latest new data?
Yes, that works too, and is probably faster, especially if you have out-of-band means for sharing read-only state between source and destination.
My question is here. If we do not have any shared storage resource between source and destination (eg. long distance VM migration), how can we migrate the latest new data to the destination? we can copy the base, mid to destination manually, then how can we migrate the active snapshot( new data goes in)? I learned that drive_mirror in qemu is built to finish this, but do not understand clearly. Could you elaborate for me, or provide an example?
Use 'virsh migrate --copy-storage-inc' to migrate only the incremental changes, which assumes that the destination can already see the same read-only backing data that the source sees.
I perform a simple test, but failed. In the source, I create: base <- mid <- active (2 snapshots, the active one is the current one) In order to migrate this vm to the destination, I manually copy both base and mid to the destination, and put them in the sam e directory /var/lib/libvirt/images/ Then I call the following commands: 'sudo virsh migrate --live --unsafe --verbose --copy-storage-inc vm1 qemu+ssh://192.168.1.3/system tcp://192.168.1.3' The error I got is: error: Requested operation is not valid: cannot migrate domain with 2 snapshots. I do not know the reason for this.
In fact, modern libvirt/qemu does this for you by setting up an NBD server on the destination, doing a data mirror from the source into the destination (so that you DO have shared storage, at least for the duration of the migration), then doing the live migration, then tearing down the NBD mirroring link.
Sounds interesting! Can you provide me an example or point me to a link about how to use this function? Sorry to take you so much time to answer my questions. I appreciate it deeply. After I solve this issue, I will write a blog and share these knowledge with other people. Best, Ed
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
[dropping multiple lists; let's just use libvirt-users] On 02/11/2015 02:45 PM, Edward Young wrote:
I perform a simple test, but failed.
In the source, I create: base <- mid <- active (2 snapshots, the active one is the current one) In order to migrate this vm to the destination, I manually copy both base and mid to the destination, and put them in the sam e directory /var/lib/libvirt/images/ Then I call the following commands: 'sudo virsh migrate --live --unsafe --verbose --copy-storage-inc vm1 qemu+ssh://192.168.1.3/system tcp://192.168.1.3'
The error I got is: error: Requested operation is not valid: cannot migrate domain with 2 snapshots.
I do not know the reason for this.
That's because libvirt is not (yet) able to migrate snapshots in a way that the destination can revert to them. It is possible to work around this in a couple of ways: 1. Manually dump all the snapshot XML configurations on the source side (virsh snapshot-list, then virsh snapshot-dumpxml in a loop), then delete the snapshot tracking (virsh snapshot-delete --metadata in a loop), then do the migration, then manually recreate the snapshot configuration on the destination (virsh snapshot-create --redefine in a loop for each of the xml files you saved earlier). If you don't plan to revert to snapshots on the destination, then you can skip the snapshot-dumpxml loop, as well as the snapshot-create --redefine loop. But if that's the case, then you can take a shortcut: 2. Don't track external snapshot data in libvirt in the first place (virsh snapshot-create --no-metadata when taking external snapshots). Instead, the backing chain is sufficient to know that external snapshot data was saved.
In fact, modern libvirt/qemu does this for you by setting up an NBD server on the destination, doing a data mirror from the source into the destination (so that you DO have shared storage, at least for the duration of the migration), then doing the live migration, then tearing down the NBD mirroring link.
Sounds interesting! Can you provide me an example or point me to a link about how to use this function?
It's all done under the hood as part of the implementation of 'virsh migrate --copy-source-*', if done as part of migration. But if you want to manually recreate the steps, the source code in src/qemu/qemu_migration.c shows the QMP commands used to set up the NBD server on the destination, then the mirroring is done with equivalent of the 'virsh blockcopy' command pointing to the NBD server as the destination.
Sorry to take you so much time to answer my questions. I appreciate it deeply. After I solve this issue, I will write a blog and share these knowledge with other people.
The libvirt-users list archives has a lot of questions and posts from other users on the same topic; it may also help if you peruse the archives. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Wed, Feb 11, 2015 at 5:09 PM, Eric Blake <eblake@redhat.com> wrote:
[dropping multiple lists; let's just use libvirt-users]
On 02/11/2015 02:45 PM, Edward Young wrote:
I perform a simple test, but failed.
In the source, I create: base <- mid <- active (2 snapshots, the active one is the current one) In order to migrate this vm to the destination, I manually copy both base and mid to the destination, and put them in the sam e directory /var/lib/libvirt/images/ Then I call the following commands: 'sudo virsh migrate --live --unsafe --verbose --copy-storage-inc vm1 qemu+ssh://192.168.1.3/system tcp://192.168.1.3'
The error I got is: error: Requested operation is not valid: cannot migrate domain with 2 snapshots.
I do not know the reason for this.
That's because libvirt is not (yet) able to migrate snapshots in a way that the destination can revert to them. It is possible to work around this in a couple of ways:
1. Manually dump all the snapshot XML configurations on the source side (virsh snapshot-list, then virsh snapshot-dumpxml in a loop), then delete the snapshot tracking (virsh snapshot-delete --metadata in a loop), then do the migration, then manually recreate the snapshot configuration on the destination (virsh snapshot-create --redefine in a loop for each of the xml files you saved earlier).
If you don't plan to revert to snapshots on the destination, then you can skip the snapshot-dumpxml loop, as well as the snapshot-create --redefine loop. But if that's the case, then you can take a shortcut:
2. Don't track external snapshot data in libvirt in the first place (virsh snapshot-create --no-metadata when taking external snapshots). Instead, the backing chain is sufficient to know that external snapshot data was saved.
It works for me. Thanks!
In fact, modern libvirt/qemu does this for you by setting up an NBD server on the destination, doing a data mirror from the source into the destination (so that you DO have shared storage, at least for the duration of the migration), then doing the live migration, then tearing down the NBD mirroring link.
Sounds interesting! Can you provide me an example or point me to a link about how to use this function?
It's all done under the hood as part of the implementation of 'virsh migrate --copy-source-*', if done as part of migration. But if you want to manually recreate the steps, the source code in src/qemu/qemu_migration.c shows the QMP commands used to set up the NBD server on the destination, then the mirroring is done with equivalent of the 'virsh blockcopy' command pointing to the NBD server as the destination.
I will take a look at the code.
Sorry to take you so much time to answer my questions. I appreciate it deeply. After I solve this issue, I will write a blog and share these knowledge with other people.
The libvirt-users list archives has a lot of questions and posts from other users on the same topic; it may also help if you peruse the archives.
I tried previously, but I can't find them. I search them directly by google? Is there anyway to find the related posts in the archive? Thanks a lot! Ed
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
participants (2)
-
Edward Young -
Eric Blake