Re: [libvirt] [PATCH 2/8] backup: Document nuances between different state capture APIs

On 06/13/2018 12:42 PM, Eric Blake wrote: This got a lot messier than originally intended. As noted in my response for .1 - I haven't really followed the discussions thus far - so take it with that viewpoint - someone from outside the current discussion trying to make sense of what this topic is all about. I would alter the sentence at the comma... IOW: In order to aid ... their needs, this page compares ... by libvirt. Then rather than discussing this below - I think we really need to state right at the top the following: </p> <p> The information here is primarily geared towards capturing the state of the active domain. Capturing state of an inactive domain essentially only requires the contents of the guest disks and then restoring that state is merely a fresh boot with the disks restored to that state. There are aspects of the subsequent functionality that cover the inactive state collection, but it's not the primary focus. to me the commas are unnecessary. s/activity./activity. In this case, state includes domain memory including the current instruction stream and domain storage, whether that is local virtual disks which are not present on a target host or networked storage being updated by the local hypervisor. A clever... [BTW: In rereading my response - I almost want to add - "As it relates to domain checkpoints and backups, state only includes disk state change.". However, I'm not sure if that ties in yet or not. I think it only matters for the two new API's being discussed.] , then there is... s/,/ in order/ [BTW: The following includes something else I pulled up from the list below...] s/had. /had. The astute reader will also realize that state capture at any level requires that the data must be stored and managed by some mechanism. This processing may be to a single file or some set of chained files. This is the inflection point between where Libvirt would (could, should?) integrate with third party tools that are built around managing the volume of data possibly generated by multiple domains with multiple disks. This leaves the task of synchronizing the capture algorithms to Libvirt in order to be able to work seamlessly with the underlying hypervisor. <paragraph break> There are several libvirt APIs associated with ... (different is superfluous) s/, such that the captured state/which s/But since ... given task./The following is a list of trade-offs and differences between the various facets that affect capturing domain state for active domains:/ "Data Completeness" (or Integrity) corresponding s/preparation (the.../preparation. The ... s/started), while .../started. While .../ Feels like a paragraph break... Or even a whole new bullet: <dt>Quiescing of Data</dt> s/around either point in time/at any point in time/ That last sentence: Freezing guest I/O can be problematic depending on what the guest's expectations are and the duration of the freeze. Some software will rightfully panic once it is given the chance to realize it lost some number of seconds. In general though long pauses are unacceptable, so reducing the time spent frozen is a goal of management software. Still the balance for data integrity and completeness does require some amount of time. I pulled part of this to earlier - let's face it, the offline guest is not the focus of this work, so I think it's only worth discussing or noting that an API can handle active or inactive guests. <dt>Memory State only, Disk change only, or Both</dt> I ended up essentially moving this concept prior to the list of facets. I suppose it could go here too, but I'm not sure this ends up being so much of a tradeoff as opposed to an overall design decision made when deciding to implement some sort of backup solution. In the long run, the management software "doesn't care" where or how the data is stored - it's just providing the data. That last sentence could almost be it's own bullet "Impact to Active State". Libvirt already captures active state as part of normal processing by updates to the domain's active XML. So again, this isn't a facet affecting capturing state - it seems to be more of a statement related to the reality of what Libvirt can/should be expected to do vs. being able to allow configurability for external forces to make certain decisions. How about Full vs. Incremental Is there a "downside" to the time needed for ma[r]king the "first checkpoint"? Or do we dictate/assume that someone has some sort of backup already before starting the domain and updates thereafter are all incremental. FWIW: A couple of others that come to mind that are facets: <dt>Local or Remote Storage</dt> Domains that completely use remote storage may only need some mechanism to keep track of the guest memory state while using some external means to manage/track the remote storage. Still even with that, it's possible that the hypervisor has I/O's "in flight" to the network storage that could be "important data" in the big picture. So having the capability to have the management software keeping track of all disk state can be important. <dt>Network Latency</dt> Whether it's domain storage or the saving of the domain data into some remote storage, network latency has an impact on snapshot data. Having dedicated network capacity/bandwidth and/or properly set quality of service certainly helps. Perhaps in some way providing an example of a migration vs. pure save could be helpful - using of course the various facets. Something like: An example of the various facets in action is migration of a running guest. In order for the guest to be able to start on a target from whence it left off on a source, the guest has to get to a point where execution on the source is stopped, the last remaining changes occurring since the migration started are then transferred, and the guest is started on the target. The management software thus must keep track of the starting point and any changes since the starting point. These last changes are often referred to as dirty page tracking or dirty disk block bitmaps. At some point in time during the migration, the management software must freeze the source guest, transfer the dirty data, and then start the guest on the target. This period of time must be minimal. To minimize overall migration time, one is advised to use a dedicated network connection with a high quality of service. Alternatively saving the current state of the running guest can just be a point in time type operation which doesn't require updating the "last vestiges" of state prior to writing out the saved state file. The state file is the point in time of whatever is current and may contain incomplete data which if used to restart the guest could cause confusion or problems because some operation wasn't completed depending upon where in time the operation was commenced. Do you think perhaps it may be a good idea to list the pros and cons of each of the APIs? As in, why someone would want to use one over another? and of course which work together... s/For qemu/Using QEMU/ Some random grumbling from me about the complexity of SnapshotCreateXML interacting with BlockCommit ;-)... Still should perhaps the Block API's be listed first to create a grounding of what they do so that when discussed as part of this Snapshot section? Of course all that without getting in the gnarly details of using Block APIs. The SnapshotCreateXML API is a "complex maze" of flag usage where it's important to understand the nuances between active/inactive, internal/external, memory/disk/both, and reversion to the point in time. s/, and/ and/ realistically the part about tracking guest memory state is probably unnecessary. s/job (to .../job. To .../ s/jobs)./jobs./ s/, with/ with/ s/), or/) or/ s/(// s/)// The next two aren't even introduced yet... But I'd probably also want to know about virDomain{Save|ManagedSave[Image]} which while not snapshot level type API's, they can be used to save domain state information. And since we discussed quiesce and freeze/thaw above, should there should virDomainFS{Freeze|Thaw} be discussed - if only to note their usage? introduced. Still for grounding this was a good introduction. I think the order should be changed since before performing an incremental backup is possible there must be some way to set when time begins. s/, and/ and/ s/, so much as make/, rather it makes/ changed? The "When performing ... current backup" description is a bit confusing to read for me. So do I create Checkpoint before after the Backup or both? Hard to be simultaneous - one comes first. This reliance on one another gets really confusing... Using the term "Guest state" for Checkpoint to mean something different than Snapshot makes for difficult comprehension in the grand scheme of domain management. Of slight concern is that there's nothing in the Checkpoint or Backup API naming scheme that says this is for disk/block only. There's also nothing that would cause me to think they are related without reading their descriptions. > + <h2><a id="examples">Examples</a></h2> > + <p>The following two sequences both capture the disk state of a > + running guest, then complete with the guest running on its > + original disk image; but with a difference that an unexpected > + interruption during the first mode leaves a temporary wrapper > + file that must be accounted for, while interruption of the > + second mode has no impact to the guest.</p> > + <p>1. Backup via temporary snapshot > + <pre> > +virDomainFSFreeze() > +virDomainSnapshotCreateXML(VIR_DOMAIN_SNAPSHOT_CREATE_DISK_ONLY) > +virDomainFSThaw() > +third-party copy the backing file to backup storage # most time spent here > +virDomainBlockCommit(VIR_DOMAIN_BLOCK_COMMIT_ACTIVE) per disk > +wait for commit ready event per disk > +virDomainBlockJobAbort() per disk > + </pre></p> > + > + <p>2. Direct backup > + <pre> > +virDomainFSFreeze() > +virDomainBackupBegin() > +virDomainFSThaw() > +wait for push mode event, or pull data over NBD # most time spent here > +virDomainBackeupEnd() > + </pre></p> > + > + </body> > +</html> The examples certainly need more beef w/r/t description. Personally, I like the fact that we're heavily documenting things before writing the code rather than the opposite direction. John