[libvirt-users] how "safe" is blockcommit ?
Hi, currently i'm following https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit. I 'm playing around with it and it seems to be quite nice. What i want is a daily consistent backup of my image file of the guest. I have the idea of the following procedure: - Shutdown the guest (i can live with a downtime of a few minutes, it will happen in the night). And i think it's the only way to have a real clean snapshot - create a snapshot with snapshot-create-as: snapshot-create-as guest testsn --disk-only - start the guest again. Changes will now go into the overlay, as e.g. inserts in a database - rsync the base file to a cifs server. With rsync not the complete, likely big file is transferred but just the delta - blockcommit the overlay: blockcommit guest /path/to/testsn --active --wait --verbose --pivot - delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata - remove the overlay Is that ok ? How "safe" is blockcommit on a running guest ? It's possible that during the rsync, when the guest is running, some inserts are done in a database. Is it safe to copy the new sectors (i assume that's what blockcommit does) under a running database ? Or is it only safe doing blockcommit on a stopped guest ? Thanks for any answer. Bernd P.S. Is the same procedure possible when the guest disk(s) reside directly in a plain logical volume, without a file system in-between ? -- Bernd Lentes Systemadministration Institut für Entwicklungsgenetik Gebäude 35.34 - Raum 208 HelmholtzZentrum münchen [ mailto:bernd.lentes@helmholtz-muenchen.de | bernd.lentes@helmholtz-muenchen.de ] phone: +49 89 3187 1241 fax: +49 89 3187 2294 [ http://www.helmholtz-muenchen.de/idg | http://www.helmholtz-muenchen.de/idg ] wer Fehler macht kann etwas lernen wer nichts macht kann auch nichts lernen Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias H. Tschoep, Heinrich Bassler, Dr. rer. nat. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671
On 09/07/2018 12:06 PM, Lentes, Bernd wrote:
Hi,
currently i'm following https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit. I 'm playing around with it and it seems to be quite nice. What i want is a daily consistent backup of my image file of the guest. I have the idea of the following procedure:
- Shutdown the guest (i can live with a downtime of a few minutes, it will happen in the night). And i think it's the only way to have a real clean snapshot
Not the only way - you can also use qemu-guest-agent with a trusted guest to quiesce all filesystem I/O after freezing database operations at a consistent point, for a clean snapshot of a live guest. But shutting down is indeed safe, and easier to reason about than worrying whether your qga interaction is properly hooked into all necessary places for a live quiesce.
- create a snapshot with snapshot-create-as: snapshot-create-as guest testsn --disk-only - start the guest again. Changes will now go into the overlay, as e.g. inserts in a database - rsync the base file to a cifs server. With rsync not the complete, likely big file is transferred but just the delta
We're also trying to add support for incremental backups into a future version of libvirt on top of the qemu 3.0 feature of persistent bitmaps in qcow2 images, which could indeed guarantee that you transfer only the portions of the guest disk that were touched since the last backup. But as that's still something I'm trying to code up, your solution of using rsync to pick out the deltas is as good as anything you can get right now.
- blockcommit the overlay: blockcommit guest /path/to/testsn --active --wait --verbose --pivot - delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata - remove the overlay
Is that ok ? How "safe" is blockcommit on a running guest ?
Yep, that's the right way to do it! It's perfectly safe - the guest doesn't care whether it is reading/writing from the backing file or the overlay, and even if the blockcommit action is aborted, you can restart it for the same effects. (Okay, if you want to get technical, you need to know that committing from 'Base <- mid <- top' down to 'Base' leaves 'mid' in an inconsistent state - but that's not something the guest can see through 'top'; and your specific case is just committing to the immediate backing layer rather than skipping layers like that).
It's possible that during the rsync, when the guest is running, some inserts are done in a database.
As long as the backing file is read-only during the rsync (which it is, since all your guest writes are going to the overlay), then nothing the guest can do will interfere with what rsync can see. Just be sure that you don't start the blockcommit until after rsync is done.
Is it safe to copy the new sectors (i assume that's what blockcommit does) under a running database ? Or is it only safe doing blockcommit on a stopped guest ?
Live blockcommit is safe, and exists so you don't have to stop the guest. For a bit more insight into what is going on under the hood, the slides from my KVM Forum talk from a few years back may give some nice insights: http://events17.linuxfoundation.org/sites/events/files/slides/2015-qcow2-exp...
Thanks for any answer.
Bernd
P.S. Is the same procedure possible when the guest disk(s) reside directly in a plain logical volume, without a file system in-between ?
Live blockcommit works onto any host storage protocol (whether filesystem, block device via LVM, or even remote access such as NBD or ceph). The key is that your overlay is a qcow2 file that is tracking the deltas during the time in which you are capturing your backup of the backing file, and then blockcommit safely writes those deltas back into the backing file prior to no longer needing the overlay. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Il 07-09-2018 21:26 Eric Blake ha scritto:
We're also trying to add support for incremental backups into a future version of libvirt on top of the qemu 3.0 feature of persistent bitmaps in qcow2 images, which could indeed guarantee that you transfer only the portions of the guest disk that were touched since the last backup. But as that's still something I'm trying to code up, your solution of using rsync to pick out the deltas is as good as anything you can get right now.
Plain rsync is going to be very slow for large file transfer. I strongly suggest to experiment with rsync "--inplace no-whole-file" options to really transfer/write changed bits only. If that does not cut in, you can try with bdsync[1] or blocksync[2] [1] https://github.com/TargetHolding/bdsync [2] https://github.com/shodanshok/blocksync
- blockcommit the overlay: blockcommit guest /path/to/testsn --active --wait --verbose --pivot - delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata - remove the overlay
Is that ok ? How "safe" is blockcommit on a running guest ?
Yep, that's the right way to do it! It's perfectly safe - the guest doesn't care whether it is reading/writing from the backing file or the overlay, and even if the blockcommit action is aborted, you can restart it for the same effects.
I found the "manual overlay rm" a bit of a pain. What is the reason why a successfull blockcommit does not delete the just-inactivate overlay file? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
On Sat, Sep 08, 2018 at 19:28:47 +0200, Gionatan Danti wrote:
Il 07-09-2018 21:26 Eric Blake ha scritto:
[...]
- blockcommit the overlay: blockcommit guest /path/to/testsn --active --wait --verbose --pivot - delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata - remove the overlay
Is that ok ? How "safe" is blockcommit on a running guest ?
Yep, that's the right way to do it! It's perfectly safe - the guest doesn't care whether it is reading/writing from the backing file or the overlay, and even if the blockcommit action is aborted, you can restart it for the same effects.
I found the "manual overlay rm" a bit of a pain. What is the reason why a successfull blockcommit does not delete the just-inactivate overlay file?
It was not implemented because currently we don't track what happens to the backing chain after a successful commit, but rather we redetect it afterwards. The commit operation can remove more than one image from the backing chain and thus tracking what happened is very important if you want to remove the files afterwards. I'm currently working on supporting -blockdev which includes full backing chain tracking so the implementation of deletion of the images obsoleted with block commit will become much simpler [*]. Peter [*] I'm not promising to implement that feature soon though.
----- On Sep 7, 2018, at 9:26 PM, Eric Blake eblake@redhat.com wrote:
On 09/07/2018 12:06 PM, Lentes, Bernd wrote:
Hi,
currently i'm following https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit. I 'm playing around with it and it seems to be quite nice. What i want is a daily consistent backup of my image file of the guest. I have the idea of the following procedure:
- Shutdown the guest (i can live with a downtime of a few minutes, it will happen in the night). And i think it's the only way to have a real clean snapshot
Not the only way - you can also use qemu-guest-agent with a trusted guest to quiesce all filesystem I/O after freezing database operations at a consistent point, for a clean snapshot of a live guest. But shutting down is indeed safe, and easier to reason about than worrying whether your qga interaction is properly hooked into all necessary places for a live quiesce.
- create a snapshot with snapshot-create-as: snapshot-create-as guest testsn --disk-only - start the guest again. Changes will now go into the overlay, as e.g. inserts in a database - rsync the base file to a cifs server. With rsync not the complete, likely big file is transferred but just the delta
We're also trying to add support for incremental backups into a future version of libvirt on top of the qemu 3.0 feature of persistent bitmaps in qcow2 images, which could indeed guarantee that you transfer only the portions of the guest disk that were touched since the last backup. But as that's still something I'm trying to code up, your solution of using rsync to pick out the deltas is as good as anything you can get right now.
- blockcommit the overlay: blockcommit guest /path/to/testsn --active --wait --verbose --pivot - delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata - remove the overlay
Is that ok ? How "safe" is blockcommit on a running guest ?
Yep, that's the right way to do it! It's perfectly safe - the guest doesn't care whether it is reading/writing from the backing file or the overlay, and even if the blockcommit action is aborted, you can restart it for the same effects.
(Okay, if you want to get technical, you need to know that committing from 'Base <- mid <- top' down to 'Base' leaves 'mid' in an inconsistent state - but that's not something the guest can see through 'top'; and your specific case is just committing to the immediate backing layer rather than skipping layers like that).
It's possible that during the rsync, when the guest is running, some inserts are done in a database.
As long as the backing file is read-only during the rsync (which it is, since all your guest writes are going to the overlay), then nothing the guest can do will interfere with what rsync can see. Just be sure that you don't start the blockcommit until after rsync is done.
Is it safe to copy the new sectors (i assume that's what blockcommit does) under a running database ? Or is it only safe doing blockcommit on a stopped guest ?
Live blockcommit is safe, and exists so you don't have to stop the guest.
For a bit more insight into what is going on under the hood, the slides from my KVM Forum talk from a few years back may give some nice insights: http://events17.linuxfoundation.org/sites/events/files/slides/2015-qcow2-exp...
Thanks for any answer.
Bernd
P.S. Is the same procedure possible when the guest disk(s) reside directly in a plain logical volume, without a file system in-between ?
Live blockcommit works onto any host storage protocol (whether filesystem, block device via LVM, or even remote access such as NBD or ceph). The key is that your overlay is a qcow2 file that is tracking the deltas during the time in which you are capturing your backup of the backing file, and then blockcommit safely writes those deltas back into the backing file prior to no longer needing the overlay.
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Hi Eric, a big thanks to this outstanding clear and thorough answer. It's a pleasure to get such information from the developers themselves, and so quick ! Bernd Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias H. Tschoep, Heinrich Bassler, Dr. rer. nat. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671
----- On Sep 7, 2018, at 9:26 PM, Eric Blake eblake@redhat.com wrote:
On 09/07/2018 12:06 PM, Lentes, Bernd wrote:
Hi,
currently i'm following https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit. I 'm playing around with it and it seems to be quite nice. What i want is a daily consistent backup of my image file of the guest. I have the idea of the following procedure:
- Shutdown the guest (i can live with a downtime of a few minutes, it will happen in the night). And i think it's the only way to have a real clean snapshot
Not the only way - you can also use qemu-guest-agent with a trusted guest to quiesce all filesystem I/O after freezing database operations at a consistent point, for a clean snapshot of a live guest. But shutting down is indeed safe, and easier to reason about than worrying whether your qga interaction is properly hooked into all necessary places for a live quiesce.
- create a snapshot with snapshot-create-as: snapshot-create-as guest testsn --disk-only - start the guest again. Changes will now go into the overlay, as e.g. inserts in a database - rsync the base file to a cifs server. With rsync not the complete, likely big file is transferred but just the delta
We're also trying to add support for incremental backups into a future version of libvirt on top of the qemu 3.0 feature of persistent bitmaps in qcow2 images, which could indeed guarantee that you transfer only the portions of the guest disk that were touched since the last backup. But as that's still something I'm trying to code up, your solution of using rsync to pick out the deltas is as good as anything you can get right now.
- blockcommit the overlay: blockcommit guest /path/to/testsn --active --wait --verbose --pivot - delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata - remove the overlay
Is that ok ? How "safe" is blockcommit on a running guest ?
Yep, that's the right way to do it! It's perfectly safe - the guest doesn't care whether it is reading/writing from the backing file or the overlay, and even if the blockcommit action is aborted, you can restart it for the same effects.
(Okay, if you want to get technical, you need to know that committing from 'Base <- mid <- top' down to 'Base' leaves 'mid' in an inconsistent state - but that's not something the guest can see through 'top'; and your specific case is just committing to the immediate backing layer rather than skipping layers like that).
It's possible that during the rsync, when the guest is running, some inserts are done in a database.
As long as the backing file is read-only during the rsync (which it is, since all your guest writes are going to the overlay), then nothing the guest can do will interfere with what rsync can see. Just be sure that you don't start the blockcommit until after rsync is done.
Is it safe to copy the new sectors (i assume that's what blockcommit does) under a running database ? Or is it only safe doing blockcommit on a stopped guest ?
Live blockcommit is safe, and exists so you don't have to stop the guest.
Hi, i thought i got the procedure i needed, but a problem arise. My guests are resources in a pacemaker cluster. I start/stop the guests through the cluster. But when i stop the guest via pacemaker, libvirt doesn't know the guest any longer. After a successfull stop "virsh list --all" doesn't show any guest. So i can't take a sanpshot with libvirt. What is about qemu-img ? Could i still use my procedure ? - stop the guest via cluster - snapshot with qemu-img - start the guest via cluster - rsync the snapshot to a file server - commit the snapshot with qemu-img to a running guest (i think this will be the problem) Thanks for any thoughts. Bernd Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: NN Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Dr. rer. nat. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671
participants (4)
-
Eric Blake -
Gionatan Danti -
Lentes, Bernd -
Peter Krempa