On Mon, Apr 30, 2018 at 08:42:56PM +0200, Gionatan Danti wrote:

Another question: how reliable are qcow2 ver2/3 files nowadays? Are you using them in production environments?

qcow2 is widely used in production at large scale in general. Just not with internal snapshots - almost everything uses external snapshots, aka backing file chains.

At the moment, I am using RAW files and filesystem-level snapshot to manage versioning; however, as virt-manager has direct support for managing qcow2 internal snapshots, it would be easier to deploy qcow2 disks.

What strikes me is that, if thing have not changed, Red Hat support policy was to *not* support internal snapshots. So, are they reliable enough for production VMs?

The QEMU community still tends to discourage use of internal snapshots. There are not even any QMP monitor commands to use them - you are forced to use the legacy HMP interface to QEMU for mgmt. All of the workaround providing interesting block storage mgmt is focused on external snapshots (aka the backing_file option). There are some technical downsides to internal snapshots IIUC, such as inability to free the space used by the internal snapshot when it is deleted, loading/saving snapshots blocks execution of the guest OS, and probably more I've forgotten about. The only nice thing about internal snapshots is simplicity of mgmt, and that is a very nice thing indeed, which is why virt-manager has code to support that - it was much easier to add that code for external snapshots. Just a shame about all the downsides :-( Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On 05/01/2018 03:11 PM, Gionatan Danti wrote:

Il 01-05-2018 10:56 Daniel P. Berrangé ha scritto:

...

qcow2 is widely used in production at large scale in general. Just not with internal snapshots - almost everything uses external snapshots, aka backing file chains.

The QEMU community still tends to discourage use of internal snapshots. There are not even any QMP monitor commands to use them - you are forced to use the legacy HMP interface to QEMU for mgmt. All of the workaround providing interesting block storage mgmt is focused on external snapshots (aka the backing_file option). There are some technical downsides to internal snapshots IIUC, such as inability to free the space used by the internal snapshot when it is deleted, loading/saving snapshots blocks execution of the guest OS, and probably more I've forgotten about.

The only nice thing about internal snapshots is simplicity of mgmt, and that is a very nice thing indeed, which is why virt-manager has code to support that - it was much easier to add that code for external snapshots. Just a shame about all the downsides :-(

So internal snapshots remain something very useful for lab/tests, but are not recommended for regular use in production environment, right?

That's fairly accurate. Also, mixing internal and external snapshots at the same time is likely to trigger some known data-loss problems, if you are not extremely careful, so for now, just pick one or the other and stick to it (I hope to someday enhance qemu to refuse operations that would risk data loss, or to perform a slower version of the operation with the same result instead of its current fast but wrong implementations, when dealing with mixed-internal/external backing chains). https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg00865.html -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org

On 01/05/2018 22:49, Gionatan Danti wrote:

Ok, I think I'll continue to use RAW images + filesystem snapshots + external snapshot when required. It is unfortunate that we have no GUI to manage external snapshots. I even remember that external snapshot remove was not supported - I had to use the --metadata flag and a plain "rm" to remove them (or using --no-metadata in the first place)[1]. Does it remain the current behavior?

Thanks.

[1] https://www.redhat.com/archives/libvirt-users/2015-September/msg00037.html

Any hope to get virsh external snapshot creation/deletion fixed? Or, even better, to have virt-manager support for external snapshots? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8

On 04/05/2018 10:26, Kashyap Chamarthy wrote:

Yes, you still have to clean up external snapshot metadata manually, as the link below indicates. Deleting external snapshots is a non-trivial problem. For example, if you have a disk image chain like:

[A] <-- [B] <-- [C] <-- [D]

Now if you no longer need image [B], what is the most efficient way to get rid of it? Three possibilities:

- Merge [B] into [A], and rewrite the backing chain of [C] to point to [A]. Now can you discard image [B]. - Merge [A] into [B], call the new image [AB]. Now you can discard [A]. - Merge [B] into [C], and rewrite the backing chain of [C] to point to [A]. Now you can discard image [B].

It depends on how large each of the disk images are, and based on the size, merge / coalesce them accordingly, to arrive at the quickest solution.

True.

Note that existing commands (`virsh blockpull` and `virsh blockcommit`, which is relatively faster) will allow you handle it manually or script it -- which many people already do it. > A couple of references (that are also easily scriptable):

http://wiki.libvirt.org/page/Live-merge-an-entire-disk-image-chain-including... https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

...

...

[1] https://www.redhat.com/archives/libvirt-users/2015-September/msg00037.html

Any hope to get virsh external snapshot creation/deletion fixed?

I don't see any problem creation external snapshots (or more correctly: "overlays"). Deletion of external snapshots involves a bit more work, but it is scriptable as well.

Reading old mailing list posts, I was under impression that some work was put to automatically and trasparently issue the best blockpull/blockcommit command. If not - fair enough ;) Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8

On 05/01/2018 04:17 PM, Paul O'Rorke wrote:

I have been using internal snapshots on production qcow2 images for a couple of years, admittedly as infrequently as possible with one exception and that exception has had multiple snapshots taken and removed using virt-manager's GUI.

I was unaware of this:

...

There are some technical downsides to internal snapshots IIUC, such as inability to free the space used by the internal snapshot when it is deleted,

This is not an insurmountable difficulty, just one that no one has spent time coding up.

This might explain why this VM recently kept going into a paused state and I had to extend the volume to get it to stay up.  This VM is used for testing our software in SharePoint and we make heavy use of snapshots.  Is there nothing I can to do recover that space?

If you have no internal snapshots, you can do a 'qemu-img convert' to copy just the portion of the image that is actively in use; the copy will use less disk space than the original because it got rid of the now-unused space. 'virt-sparsify' from libguestfs takes this one step further, by also removing unused space within the guest filesystem itself. In fact, even if you do have internal snapshots, there is probably a sequence of 'qemu-img convert' invocations that can ultimately convert all of your internal snapshots into an external chain of snapshots; but I don't have a ready formula off-hand to point to (experiment on an image you don't care about, before doing it on your production image).

What would be the best practice then for a VM that needs to be able to create and remove snapshots on a regular basis?

In a managed environment, external snapshots probably have the most support for creating and later merging portions of a chain of snapshots, although we could still improve libvirt to make this feel like more of a first class citizen. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org

Thanks for the suggestions Eric, great stuff and most appreciated. I must play in my sandbox... *Paul O'Rorke* *Tracker Software Products (Canada) Limited * www.tracker-software.com <http://www.tracker-software.com/> Tel: +1 (250) 324 1621 Fax: +1 (250) 324 1623 <http://www.tracker-software.com/> Support: http://www.tracker-software.com/support Download latest Releases http://www.tracker-software.com/downloads/ On 2018-05-01 02:31 PM, Eric Blake wrote:

On 05/01/2018 04:17 PM, Paul O'Rorke wrote:

...

I have been using internal snapshots on production qcow2 images for a couple of years, admittedly as infrequently as possible with one exception and that exception has had multiple snapshots taken and removed using virt-manager's GUI.

I was unaware of this:

...

There are some technical downsides to internal snapshots IIUC, such as inability to free the space used by the internal snapshot when it is deleted,

This is not an insurmountable difficulty, just one that no one has spent time coding up.

...

This might explain why this VM recently kept going into a paused state and I had to extend the volume to get it to stay up.  This VM is used for testing our software in SharePoint and we make heavy use of snapshots.  Is there nothing I can to do recover that space?

If you have no internal snapshots, you can do a 'qemu-img convert' to copy just the portion of the image that is actively in use; the copy will use less disk space than the original because it got rid of the now-unused space.  'virt-sparsify' from libguestfs takes this one step further, by also removing unused space within the guest filesystem itself.

In fact, even if you do have internal snapshots, there is probably a sequence of 'qemu-img convert' invocations that can ultimately convert all of your internal snapshots into an external chain of snapshots; but I don't have a ready formula off-hand to point to (experiment on an image you don't care about, before doing it on your production image).

...

What would be the best practice then for a VM that needs to be able to create and remove snapshots on a regular basis?

In a managed environment, external snapshots probably have the most support for creating and later merging portions of a chain of snapshots, although we could still improve libvirt to make this feel like more of a first class citizen.

On 05/22/2018 01:53 PM, Paul O'Rorke wrote:

Hi Eric and list,

I had another production VM start pausing itself.  This one had been running for more than 4 years on a 60G LVM volume.  It has had the occasional snapshot during that time though all have been "removed" using the virt-manager gui so I used qemu-img as you suggested.

# qemu-img convert /dev/trk-kvm-02-vg/rt44 -O qcow2 /mnt/scratch/rt44.qcow2

To make sure I understand, /dev/trk-kvm-02-vg/rt44 was using qcow2 on top of a block device, both before and after you used qemu-img convert to compact out the wasted space? Also, do you know if you were using qcow2v2 or v3 prior to running out of space? qcow2v2 was the default in CentOS 6 for historical reasons, but qemu doesn't support efficient space reclamation on those older images the way it does on qcow2v3. Did you save the full image anywhere, so we can do additional postmortem analysis on it? Not that I'm asking to see the image itself, in case that would give away private information; but even something like running 'qcheck' from https://github.com/jnsnow/qcheck.git might shed some insight into the storage being used by the image that reached capacity.

dd'd the qcow2 image back on to the LV after testing it boots OK directly from the image and it is in production again.

The VM itself reports ample space available:

$ df -h Filesystem                       Size  Used Avail Use% Mounted on udev                             3.9G     0  3.9G   0% /dev tmpfs                            789M  8.8M  780M   2% /run /dev/mapper/RT--vg-root           51G   21G   28G  42% / tmpfs                            3.9G     0  3.9G   0% /dev/shm tmpfs                            5.0M     0  5.0M   0% /run/lock tmpfs                            3.9G     0  3.9G   0% /sys/fs/cgroup /dev/vda1                        472M  155M  293M  35% /boot 192.168.0.16:/volume1/fileLevel  8.1T  2.5T  5.6T  31% /mnt/nfs/fileLevel tmpfs                            789M     0  789M   0% /run/user/1000

I would prefer to not get caught out again with this machine pausing, how can I determine how much space is being used up by 'deleted' internal snapshots?  Do you have any suggested reading on this?

The highest write offset (block.<num>.allocation in virConnectGetAllDomainStats()) should tell you how much of the underlying block device is "in use", that is, the highest offset that is being written to. It doesn't count for holes earlier in the image, especially where space could be reclaimed by reusing those holes, but does give a good indication when you might need to resize a storage volume to account for growing amounts of metadata that aren't being cleared. And it may be that qemu still needs some patches to more fully trim and reuse the space previously occupied by a deleted internal snapshot.

If I extend the LVM volume but not the guest file system will snapshots be "at the end" of the LV and "outside" the guest file system?

qcow2 is a mapped file format. Resizing the LVM volume does NOT change the amount of disk space seen by the guest. Clusters may appear in a different order in the host's raw storage than the order in which they are visited in the guest ('qemu-img map' can show you the mapping), but the guest does not care because it is always presented a logical view of a linear array of storage, regardless of how clusters are fragmented across the host device. Right now, qcow2 does not have any way to constrain the clusters used by guest data to appear at any particular offset (although there has been talk of adding a new mode to qemu to operate on a fully-preallocated image where all guest clusters except maybe the first occur in linear order at the beginning of the file, and all qcow2 metadata except for the leading header be placed at the end of the file at offsets higher than the guest clusters - in such a layout, as long as you don't use internal snapshots, then all further qcow2 metadata writes would be beyond the linear region reserved for guest-visible clusters).

If I were to expand the guest's ext4 file system I would want to do it unmounted and from a live CD but I'm having a heck of a time getting my live distro to use the virtio disk drivers.  Any advice there?

virt-resize from libguestfs-tools is your friend! -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org

On Tue, May 1, 2018 at 10:45 AM, Daniel P. Berrangé <berrange@redhat.com> wrote:

On Tue, Jan 30, 2018 at 01:17:21PM +0100, Gionatan Danti wrote:

...

Hi all, on a fully patched CentOS 7.4 x86-64, I see the following behavior:

- when creating a new volumes using vol-create-as, the resulting file is a qcow2 version 2 (compat=0.10) file. Example:

[root@gdanti-lenovo vmimages]# virsh vol-create-as default zzz.qcow2 8589934592 --format=qcow2 --backing-vol /mnt/vmimages/centos6.img Vol zzz.qcow2 created

Yes, for sake of backcompat we default to v2 unless something requires v3. You can't use vol-create-as to create a v3 image, you need to use the XML input.

BTW: in virt-manager of CentOS 7.4 by default qcow2 disks (see below for zzz3.qcow2) are created with compatibility 1.1 (probably indirectly because of its lazy refcounts feature enabled, as Daniel wrote): [root@c7client images]# qemu-img info zzz3.qcow2 image: zzz3.qcow2 file format: qcow2 virtual size: 1.0G (1073741824 bytes) disk size: 516K cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: true [root@c7client images]# The same if I create a volume with a backing file from virt-manager (not important if this is 0.10 or 1.1): [root@c7client images]# qemu-img info zzz4.qcow2 image: zzz4.qcow2 file format: qcow2 virtual size: 1.0G (1073741824 bytes) disk size: 196K cluster_size: 65536 backing file: /var/lib/libvirt/images/zzz2.qcow2 backing file format: qcow2 Format specific information: compat: 1.1 lazy refcounts: true [root@c7client images]# But I don't know if internally virt-manager uses virsh or qemu-img actually.... My tests with: libvirt-client-3.2.0-14.el7_4.9.x86_64 virt-manager-1.4.1-7.el7.noarch The same default to 0.10 format seems true with virsh from Fedora 27 and libvirt-client-3.7.0-4.fc27.x86_64 Gianluca