On 4/26/24 16:50, Daniel P. Berrangé wrote:
On Fri, Apr 26, 2024 at 11:44:38AM -0300, Fabiano Rosas wrote:
> Daniel P. Berrangé <berrange(a)redhat.com> writes:
>
>> On Fri, Apr 26, 2024 at 10:03:29AM -0300, Fabiano Rosas wrote:
>>> Daniel P. Berrangé <berrange(a)redhat.com> writes:
>>>
>>>> On Wed, Apr 17, 2024 at 05:12:27PM -0600, Jim Fehlig via Devel wrote:
>>>>> A good starting point on this journey is supporting the new
mapped-ram
>>>>> capability in qemu 9.0 [2]. Since mapped-ram is a new on-disk format,
I
>>>>> assume we'll need a new QEMU_SAVE_VERSION 3 when using it?
Otherwise I'm not
>>>>> sure how to detect if a saved image is in mapped-ram format vs the
existing,
>>>>> sequential stream format.
>>>>
>>>> Yes, we'll need to be supporting 'mapped-ram', so a good
first step.
>>>>
>>>> A question is whether we make that feature mandatory for all save
images,
>>>> or implied by another feature (parallel save), or an directly
controllable
>>>> feature with opt-in.
>>>>
>>>> The former breaks back compat with existnig libvirt, while the latter 2
>>>> options are net new so don't have compat implications.
>>>>
>>>> In terms of actual data blocks written on disk mapped-ram should be be
the
>>>> same size, or smaller, than the existing format.
>>>>
>>>> In terms of logical file size, however, mapped-ram will almost always be
>>>> larger.
>>>>
>>>> This is because mapped-ram will result in a file whose logical size
matches
>>>> the guest RAM size, plus some header overhead, while being sparse so not
>>>> all blocks are written.
>>>>
>>>> If tools handling save images aren't sparse-aware this could come
across
>>>> as a surprise and even be considered a regression.
>>>>
>>>> Mapped ram is needed for parallel saves since it lets each thread write
>>>> to a specific region of the file.
>>>>
>>>> Mapped ram is good for non-parallel saves too though, because the
mapping
>>>> of RAM into the file is aligned suitably to allow for O_DIRECT to be
used.
>>>> Currently libvirt has to tunnnel over its iohelper to futz alignment
>>>> needed for O_DIRECT. This makes it desirable to use in general, but back
>>>> compat hurts...
>>>
>>> Note that QEMU doesn't support O_DIRECT without multifd.
>>>
>>> From mapped-ram patch series v4:
>>>
>>> - Dropped support for direct-io with fixed-ram _without_ multifd. This
>>> is something I said I would do for this version, but I had to drop
>>> it because performance is really bad. I think the single-threaded
>>> precopy code cannot cope with the extra latency/synchronicity of
>>> O_DIRECT.
>>
>> Note the reason for using O_DIRECT is *not* to make saving / restoring
>> the guest VM faster. Rather it is to ensure that saving/restoring a VM
>> does not trash the host I/O / buffer cache, which will negatively impact
>> performance of all the *other* concurrently running VMs.
You can absolutely also thrash yourself, not only other VMs.
>
> Well, there's surely a performance degradation threshold that negates
> the benefits of perserving the caches. But maybe it's not as low as I
> initially thought then.
I guess you could say that O_DIRECT makes saving/restoring have a
predictable speed, because it will no longer randomly vary depending
on how much free RAM happens to be available at a given time. Time
will be dominated largely by the underlying storage I/O performanc
With fast nvme
disks, my observation is that O_DIRECT without multifd is bottlenecked on libvirt + QEMU
throughput, not on storage I/O Performance.
With regards,
Daniel
Ciao,
Claudio