Re: [PATCH 0/3] Implement support for QCOW2 data files

On Mon, Aug 12, 2024 at 12:04:01 +0200, Denis V. Lunev wrote: > On 8/12/24 10:36, Peter Krempa wrote: >> On Mon, Aug 12, 2024 at 09:26:08 +0200, Peter Krempa wrote: >>> On Sun, Aug 11, 2024 at 17:34:45 +0300, Nikolai Barybin via Devel wrote: >>>> There are use cases when the existing disks (i.e. LVM) are wanted >>>> to be used with advanced features. For this purpose QEMU allows >>>> data-file feature for qcow2 files: metadata is kept in the qcow2 >>>> file like usual, but guest data is written to an external file. >>>> These patches enable support for this feature in libvirt. >>> So this feature was once attempted to be added but was never finished >>> and the comitted bits were reverted eventually. (I've purged my local >>> archive so I don't have the link handy but I can look if you want the >>> links to the old posting) >>> >>> It was deemed that this doesn't really add any performance benefit over >>> storing the actual qcow2 with data inside. The qcow2 with data can be >>> stored inside the LV or other block device for that matter and thus can >>> provide all features that are necessary. The data file feature makes >>> also the management of the metadata and data much more complex, for a >>> very bad trade-off. At this point with 'qemu-img measure' it's easy to >>> query the necessary size to have a fully allocated qcow2 inside the >>> block device. >>> >>> Based on the history of this I'd like to ask you to summarize >>> justifications and reasons for adding this feature before continuing. >> Based on the current state of the series and what would be required to >> make it viable to be accepted I very strongly suggest re-thinking if you >> really need this feature, especially based on the caveats above. >> > Let me clarify a bit. > > QCOW2 as a data file uses QCOW2 as a metadata storage for ordinary > block devices. This is a feature of QEMU and it would be quite > natural to have its representation in the libvirt as without > libvirt help QEMU could not even start with such a configuration > due to namespaces. > > LVM or not LVM. How it is better in comparison with the QCOW2. Historically when this was considered to be used for the incremental backup feature in oVirt, similar set of the advantages was picked as the justification. Later on after discussing this for a bit it became obvious that the advantages are not as great to justify the extra effort: - extra management complexity (need to carry over the qcow2 as well as the data file) - possibility to desynchronize the state (write into the 'data_file' invalidates the qcow2 "overlay"), without being able to see that it was desync'd (writing into the data file doesn't touch the overlay so the qcow2 driver doesn't see that). - basically no performance benefits on top of qcow2 - (perhaps other's on oVirt side ... it was long time ago so I don't remember any more)

> First of all, there are historical setups when the customer > uses LVM for virtual machines and does not want to reformat > his hard disks to the file system. This makes a sense as we Yes, such setups would not be possible. Users would need to convert to qcow2 but that can be done transparently to the VM. (but briefly requires twice the storage).

are avoiding two level of metadata, i.e. QCOW2 metadata over > local FS metadata. This makes the setup a little bit more As stated above you don't really need to use a filesystem. You can make a block device (whether real/partition/lv) into a qcow2 image by simply using qemu-img format.

reliable. It should also be noted that QCOW2 in very specific > setups is at least 2 times slow (when L2 cache does not fit > into the dedicated amount of RAM while the disk itself is > quite big. I would admit that this problem would be seen even > for not so big 1T disks). Doesn't a QCOW2 with a 'data file' behave the same as a fully-allocated qcow2? I don't really see how this is more reliable/performant than plain fully-allocated qcow2.

> On top of that LVM is quite useful once we start talking about > clustered LVM setups. Clustered LVM is a good alternative > for CEPH at least for some users. > > Thus this data file setup is a way to provide backups, VM > state snapshots and other features. Backups (changed block tracking) is the only thing you'd gain with this. Snapshots and other features are already possible with external snapshots. While based on the above I don't really see the need/use for 'data_file' qcow2 feature, especially given the complexity of adding it properly, I'm not opposed as long as it will be implemented properly. I suggest having a look at Cole's patches from the last attempt as they were far more complete than this posting.