On Thu, Apr 25, 2024 at 04:41:02PM -0600, Jim Fehlig via Devel wrote: The issue with that is the semantics around old libivrt loading the new image. Old libvirt won't know to look for 'mapped-ram' element/attribute in the XML cookie, so will think it is a traditional image with hilariously predictable results :-) Hence I think mapped-ram needs some addition/change to the save image header that explicitly stops old libvirt trying to consume it, thus the version bump you mention. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Wed, Apr 17, 2024 at 05:12:27PM -0600, Jim Fehlig via Devel wrote: Yes, we'll need to be supporting 'mapped-ram', so a good first step. A question is whether we make that feature mandatory for all save images, or implied by another feature (parallel save), or an directly controllable feature with opt-in. The former breaks back compat with existnig libvirt, while the latter 2 options are net new so don't have compat implications. In terms of actual data blocks written on disk mapped-ram should be be the same size, or smaller, than the existing format. In terms of logical file size, however, mapped-ram will almost always be larger. This is because mapped-ram will result in a file whose logical size matches the guest RAM size, plus some header overhead, while being sparse so not all blocks are written. If tools handling save images aren't sparse-aware this could come across as a surprise and even be considered a regression. Mapped ram is needed for parallel saves since it lets each thread write to a specific region of the file. Mapped ram is good for non-parallel saves too though, because the mapping of RAM into the file is aligned suitably to allow for O_DIRECT to be used. Currently libvirt has to tunnnel over its iohelper to futz alignment needed for O_DIRECT. This makes it desirable to use in general, but back compat hurts... Looking at what we did in the past First time, we stole a element from 'uint32_t unused[..]' in the save header, to add the 'compressed' field, and bumped the version. This prevented old libvirt reading the files. This was needed as adding compression was a non-backwards compatible change. We could have carried on using version 1 for non-compressd fields, but we didn't for some reason. It was a hard compat break. Next time, we stole a element from 'uint32 unused[..]' in the save header, to add the 'cookie_len' field, but did NOT bump the version. 'unused' is always all zeroes, so new libvirt could detect whether the cookie was present by the len being non-zero. Old libvirt would still load the image, but would be ignoring the cookie data. This was largely harmless. This time mapped-ram is a non-compatible change, so we need to ensure old libvirt won't try to read the files, which suggests either a save version bump, or we could abuse the 'compressed' field to indicate 'mapped-ram' as a form of compression. If we did a save version bump, we might want to carrry on using v2 for non mapped ram. 'mapped-ram' is reported in QMP as a MigrationCapability, so I think we can probe for it directly. Yes, it is exclusively for use with 'file:' protocol. If we want to use FD passing, then we can still do that with 'file:', by using QEMU's generic /dev/fdset/NNN approach we have with block devices. One option is to be lazy and have a /etc/libvirt/qemu.conf for the save format version, defaulting to latest v3. Release note that admin/host provisioning apps must set it to v2 if back compat is needed with old libvirt. If we assume new -> old save image loading is relatively rare, that's probably good enough. IOW, we can * Bump save version to 3 * Use v3 by default * Add a SAVE_PARALLEL flag which implies mapped-ram, reject if v2 * Use mapped RAM with BYPASS_CACHE for v3, old approach for v2 * Steal another unused field to indicate use of mapped-ram, or perhaps future proof it by declaring a 'features' field. So we don't need to bump version again, just make sure that the libvirt loading an image supports all set features. Yes, mapped-ram should be a pre-requisite. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Apr 26, 2024 at 10:03:29AM -0300, Fabiano Rosas wrote: Note the reason for using O_DIRECT is *not* to make saving / restoring the guest VM faster. Rather it is to ensure that saving/restoring a VM does not trash the host I/O / buffer cache, which will negatively impact performance of all the *other* concurrently running VMs. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Apr 26, 2024 at 11:44:38AM -0300, Fabiano Rosas wrote: I guess you could say that O_DIRECT makes saving/restoring have a predictable speed, because it will no longer randomly vary depending on how much free RAM happens to be available at a given time. Time will be dominated largely by the underlying storage I/O performance With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On 4/26/24 4:04 AM, Daniel P. Berrangé wrote: It feels more like an implementation detail. Correct. E.g. from a mostly idle 8G VM # stat existing-format.sav Size: 510046983 Blocks: 996192 IO Block: 4096 regular file # stat mapped-ram-format.sav Size: 8597730739 Blocks: 956200 IO Block: 4096 regular file The upside is mapped-ram is bounded, unlike the existing stream, which can result in actual file sizes much greater than RAM size when VM runs a memory intensive workload. Yes, I already had visions of phone ringing off the hook asking "why are my save images suddenly huge?". But maybe it's tolerable once they realize actual blocks used, and when combined with parallel they could also be asking "why are saves suddenly so fast?" :-). My POC avoids the use of iohelper with mapped-ram. It provides qemu with two fds when direct-io has been requested, one opened with O_DIRECT, one without. Yes, mapped-ram is reported. Sorry for not being clear, but I was asking if qemu advertised support for the 'file:' migration URI it gained in 8.2? Probably not a problem either way since it predates mapped-ram. This sounds like a reasonable start. Thanks for the feedback. Regards, Jim