Re: Revisiting parallel save/restore
On 4/17/24 5:12 PM, Jim Fehlig wrote:
Hi All,
While Fabiano has been working on improving save/restore performance in qemu,
I've been tinkering with the same in libvirt. The end goal is to introduce a new
VIR_DOMAIN_SAVE_PARALLEL flag for save/restore, along with a
VIR_DOMAIN_SAVE_PARAM_PARALLEL_CONNECTIONS parameter to specify the number of
concurrent channels used for the save/restore. Recall Claudio previously posted
a patch series implementing parallel save/restore completely in libvirt, using
qemu's multifd functionality [1].
A good starting point on this journey is supporting the new mapped-ram
capability in qemu 9.0 [2]. Since mapped-ram is a new on-disk format, I assume
we'll need a new QEMU_SAVE_VERSION 3 when using it? Otherwise I'm not sure how
to detect if a saved image is in mapped-ram format vs the existing, sequential
stream format.
While hacking on a POC, I discovered the save data cookie and assume the use of
mapped-ram could be recorded there?
IIUC, mapped-ram cannot be used with the exiting 'fd:'
migration URI and instead
must use 'file:'. Does qemu advertise support for that? I couldn't find it.
If
not, 'file:' (available in qemu 8.2) predates mapped-ram, so in theory we could
live without the advertisement.
It's also not clear when we want to enable the mapped-ram capability. Should it
always be enabled if supported by the underlying qemu? One motivation for
creating the mapped-ram was to support direct-io of the migration stream in
qemu, in which case it could be tied to VIR_DOMAIN_SAVE_BYPASS_CACHE. E.g. the
mapped-ram capability is enabled when user specifies
VIR_DOMAIN_SAVE_BYPASS_CACHE && user-provided path results in a seekable fd
&&
qemu supports mapped-ram?
Comments/suggestions on these topics are much appreciated :-).
Looking ahead, should the mapped-ram capability be required for
supporting the
VIR_DOMAIN_SAVE_PARALLEL flag?
I think the answer is yes, otherwise we'd need something in libvirt like
Claudio's original series to manage multifd channels writing to fixed offsets in
the save file.
Regards,
Jim
As I understand, parallel save/restore was
another motivation for creating the mapped-ram feature. It allows multifd
threads to write exclusively to the offsets provided by mapped-ram. Can multiple
multifd threads concurrently write to an fd without mapped-ram?
Regards,
Jim
[1]
https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/3Y...
[2]
https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/migration/m...