Re: [libvirt RFC] add API for parallel Saves (not for committing)

On 4/21/22 7:08 PM, Daniel P. Berrangé wrote: > On Thu, Apr 14, 2022 at 09:54:16AM +0200, Claudio Fontana wrote: >> RFC, starting point for discussion. >> >> Sketch API changes to allow parallel Saves, and open up >> and implementation for QEMU to leverage multifd migration to files, >> with optional multifd compression. >> >> This allows to improve save times for huge VMs. >> >> The idea is to issue commands like: >> >> virsh save domain /path/savevm --parallel --parallel-connections 2 >> >> and have libvirt start a multifd migration to: >> >> /path/savevm : main migration connection >> /path/savevm.1 : multifd channel 1 >> /path/savevm.2 : multifd channel 2 > > At a conceptual level the idea would to still have a single file, > but have threads writing to different regions of it. I don't think > that's possible with multifd though, as it doesn't partition RAM > up between threads, its just hands out pages on demand. So if one > thread happens to be quicker it'll send more RAM than another > thread. Also we're basically capturing the migration RAM, and the > multifd channels have control info, in addition to the RAM pages. > > That makes me wonder actually, are the multifd streams unidirectional > or bidirectional ? Our saving to a file logic, relies on the streams > being unidirectional. Unidirectional. In the meantime I completed an actual libvirt prototype that works (only did the save part, not the restore yet). > > You've got me thinking, however, whether we can take QEMU out of > the loop entirely for saving RAM. > > IIUC with 'x-ignore-shared' migration capability QEMU will skip > saving of RAM region entirely (well technically any region marked > as 'shared', which I guess can cover more things). Heh I have no idea about this. > > If the QEMU process is configured with a file backed shared > memory, or memfd, I wonder if we can take advantage of this. > eg > > 1. pause the VM > 1. write the libvirt header to save.img > 2. sendfile(qemus-memfd, save.img-fd) to copy the entire > RAM after header I don't understand this point very much... if the ram is already backed by file why are we sending this again..?

> 3. QMP migrate with x-ignore-shared to copy device > state after RAM > > Probably can do the same on restore too. Do I understand correctly that you suggest to constantly update the RAM to file at runtime? Given the compute nature of the workload, I'd think this would slow things down.