2013/3/12 Nicolas Sebrecht <nsebrecht(a)piing.fr&gt; far as I know if a windows2003 directly poweroff (not normal shutdown),then this window will open when bootup.

On 03/12/2013 07:35 AM, Nicolas Sebrecht wrote: That shouldn't be happening - a start after a managedsave should be restoring the guest to the same state as at the save. It sounds like you may have run into a corrupted managedsave file, so libvirt punted and booted the guest from scratch instead of restoring state; booting from scratch without a clean shutdown would explain the symptoms of the OS complaining. Did you upgrade qemu in between when you saved your guest and restarted it? If so, this may be more of a qemu bug about not handling incoming migration of data generated from an older qemu. Are you sure that the managed save data was not corrupted, such as a power outage occurring before the managed save file was completely flushed to disk? -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

On 03/13/2013 09:25 AM, Nicolas Sebrecht wrote: You might want to look into external snapshots as a more efficient way of taking guest snapshots. With virsh managedsave/start, you are guaranteed guest downtime of several seconds at a minimum, and you have to do the disk management yourself; but with external snapshots, you can take a consistent picture of guest state at a point in time with sub-second downtime (the same way live migration between hosts has sub-second downtime). -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

On 03/14/2013 06:29 AM, Nicolas Sebrecht wrote: Do be aware that an external snapshot means you are no longer using a raw image - it forces you to use a qcow2 file that wraps your raw image. With new enough libvirt and qemu, it is also possible to use 'virsh blockcopy' instead of snapshots as a backup mechanism, and THAT works with raw images without forcing your VM to use qcow2. But right now, it only works with transient guests (getting it to work for persistent guests requires a persistent bitmap feature that has been proposed for qemu 1.5, along with more libvirt work to take advantage of persistent bitmaps). There's also a proposal on the qemu lists to add a block-backup job, which I would need to expose in libvirt, which has even nicer backup semantics than blockcopy, and does not need a persistent bitmap. Specify an absolute patch, not a relative one. Actually, it would probably be better if libvirtd errored out on relative path names (relative to what? libvirtd runs in '/', and has no idea what directory virsh was running in), and therefore virsh should be nice and convert names to absolute before handing them to libvirtd. Correct - we still don't have 'snapshot-revert' wired up in libvirt to revert to an external snapshot - we have ideas on what needs to happen, but it will take time to get that code into the code base. So for now, you have to do that manually. Hmm. This restored the memory state from the point at which the snapshot was taken, but unless you were careful to check that the saved state referred to the base file name and not the just-created qcow2 wrapper from when you took the snapshot, then your disks might be in an inconsistent state with the memory you are loading. Not good. Also, restoring from the base image means that you are invalidating the contents of the qcow2 file for everything that took place after the snapshot was taken. Yeah, again a known limitation. Once you change state behind libvirt's back (because libvirt doesn't yet have snapshot-revert wired up to do things properly), you generally have to 'virsh snapshot-delete --metadata VM snap1' to tell libvirt to forget the snapshot existed, but without trying to delete any files, since you did the file deletion manually. You have to delete /VM.save and /VM-snap1.img yourself, but you should have used 'virsh snapshot-delete --metadata' instead of mucking around in /var/lib (that directory should not be managed manually). Here, qemu is at fault. They have not yet given us a command to do that with minimal downtime. They HAVE given us 'virsh blockcommit', but it is currently limited to reducing chains of length 3 or longer to chains of at least 2. It IS possible to merge back into a single file while the guest remains live, by using 'virsh blockpull', but that single file will end up being qcow2; and it takes the time proportional to the size of the entire disk, rather than to the size of the changes since the snapshot was taken. Again, here's hoping that qemu 1.5 gives us live commit support, for taking a chain of 2 down to a single raw image. You went behind libvirt's back and removed /VM-snap1.img, but failed to update the managedsave image to record the location of the new filename. Yes, but you still have the managedsave image in the way. Try 'virsh managedsave-remove VM' to get the broken managedsave image out of the way. Or, if you are brave, and insist on rebooting from the memory state at which the managedsave image was taken but are sure you have tweaked the disks correctly to match the same point in time, then you can use 'virsh save-image-edit /path/to/managedsave' (assuming you know where to look in /etc to find where the managedsave file was stored internally by libvirt). Since modifying files in /etc is not typically recommended, I will assume that if you can find the right file to edit, you are already brave enough to take on the consequences of going behind libvirt's back. At any rate, editing the managed save image to point back to the correct raw file name, followed by 'virsh start', will let you resume with memory restored to the point of your managed save (and hopefully you pointed the disks to the same point in time). Aha - so you ARE brave, and DID try to edit the managedsave file. I'm surprised that you really hit a case where your edits pushed the XML over a 4096-byte boundary. Can you come up with a way to (temporarily) use shorter names, such as having /VM-snap1.img be a symlink to the real file, just long enough for you to get the domain booted again? Also, I hope that you did your experimentation on a throwaway VM, and not on a production one, in case you did manage to fubar things to the point of data corruption by mismatching disk state vs. memory state. Yes, I know that reverting to snapshots is still very much a work in progress in libvirt, and that you are not the first to ask these sorts of questions (reading the list archives will show that this topic comes up quite frequently). -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

On 03/15/2013 06:17 AM, Nicolas Sebrecht wrote: This involves guest downtime, longer according to how much state changed since the snapshot. Guests can be made temporarily transient. That is, the following sequence has absolute minimal guest downtime, and can be done without any qcow2 files in the mix. For a guest with a single disk, there is ZERO! downtime: virsh dumpxml --security-info dom > dom.xml virsh undefine dom virsh blockcopy dom vda /path/to/backup --wait --verbose --finish virsh define dom.xml For a guest with multiple disks, the downtime can be sub-second, if you script things correctly (the downtime lasts for the duration between the suspend and resume, but the steps done in that time are all fast): virsh dumpxml --security-info dom > dom.xml virsh undefine dom virsh blockcopy dom vda /path/to/backup-vda virsh blockcopy dom vdb /path/to/backup-vdb polling loop - check periodically until 'virsh blockjob dom vda' and 'virsh blockjob dom vdb' both show 100% completion virsh suspend dom virsh blockjob dom vda --abort virsh blockjob dom vdb --abort virsh resume dom virsh define dom.xml In other words, 'blockcopy' is my current preferred method of online guest backup, even though I'm still waiting for qemu improvements to make it even nicer. Not if you don't mind being temporarily transient. For that, I will probably be adding a 'virsh blockbackup dom vda' command. virsh would still support relative paths, it's just the underlying libvirtd should require absolute (in other words, the UI should do the normalizing, so that the RPC is unambiguous; right now the RPC is doing the normalization, but to the wrong directory because it doesn't know what the wording directory of the UI is). This is a bug, but easy enough to fix, and in the meantime, easy enough for you to work around (use absolute instead of relative, until libvirt 1.0.4 is out). During this step, the qcow2 files created in step 1 are getting larger proportional to the amount of changes done in the guest; obviously, the faster you can complete it, the smaller the deltas will be, and the faster your later merge steps will be. Since later merge steps have to be done while the guest is halted, it's good to keep small size in mind. More on this thought below... By 'halt the vm state', do you mean power it down, so that you would be doing a fresh boot (aka 'virsh shutdown dom', do your work including 'virsh edit dom', 'virsh start dom')? Or do you mean 'take yet another snapshot', so that you stop qemu, manipulate things to point to the right files, then start a new qemu but pickup up at the same point where the running guest left off (aka 'virsh save dom file', do your work including 'virsh save-file-edit file', 'virsh restore file')? My advice: Don't use managedsave. At this point, it just adds more confusion, and you are better off directly using 'virsh save' (managedsave is just a special case of 'virsh save', where libvirt picks the file name on your behalf, and where 'virsh start' is smart enough to behave like 'virsh restore file' on that managed name - but that extra magic in 'virsh start' makes life that much harder for you to modify what the guest will start with). This step is done with raw qemu-img commands at the moment, and takes time proportional to the size of the qcow2 data. Based on how you stopped the vm in step 3, this is either via 'virsh start' (assuming you did 'virsh shutdown', or via 'virsh restore' (assuming you did 'virsh save'; with the special case that if you did 'virsh managedsave, 'virsh start' behaves like 'virsh restore'). ...As mentioned above, the time taken in step 2 can affect how big the delta is, and therefore how long step 4 lasts (while the guest is offline). If your original disk is huge, and copying it to your backup takes a long amount of time, it may pay to do an iterative approach: start with raw image: raw.img create the external snapshot at the point you care about raw.img <- snap1.qcow2 transfer raw.img and vmstate file to backup storage, taking as long as needed (gigabytes of data, so on the order of minutes, during which the qcow2 files can build up to megabytes in size) raw.img <- snap1.qcow2 create another external snapshot, but this time with --disk-only --no-metadata (we don't plan on reverting to this point in time) raw.img <- snap1.qcow2 <- snap2.qcow2 use 'virsh blockcommit dom vda --base /path/to/raw --top /path/to/snap1 --wait --verbose'; this takes time for megabytes of storage, but not gigabytes, so it is faster than the time to copy raw.img, which means snap2.qcow2 will hold less delta data than snap1.qcow2 raw.img <- snap2.qcow2 now stop the guest, commit snap2.qcow2 into raw.img, and restart the guest By doing an iteration, you've reduced the size of the file that has to be committed while the guest is offline; and may be able to achieve a noticeable reduction in guest downtime. Here, 'virsh destroy' is fine, if you don't care about maintaining parallel branches of execution past the snapshot point. If you DO plan on toggling between two parallel branches from a common snapshot, be sure to take another snapshot at this point in time. Here, you need to use 'virsh restore' on the file that holds the vm state from the point of the snapshot. You can double-check what it points to with 'virsh save-image-dumpxml', to make sure. In fact, if you KNOW you don't care about libvirt tracking snapshots, you can do 'virsh snapshot-create[-as] --no-metadata dom ...' in the first place, so that you get the side effects of external file creation without any of the (soon-to-be-useless) metadata in the first place. Again, my recommendation is to NOT use managedsave. It changes what 'virsh start' will do: if there is a managedsave image present, it takes precedence over anything you do in 'virsh edit', unless you use 'virsh start --force-boot' to intentionally discard the managedsave image. On the other hand, since the managedsave image is the only record of the running vm state, you don't want it discarded, which means your attempt to use 'virsh edit' are useless, and you are forced to use 'virsh save-image-edit' on a file that should be internal to libvirt. > Sounds like we discussed this up above - you can either do this offline (virsh shutdown, edit, virsh start) or on a running image (virsh save, edit, virsh restore). Yeah, being more careful about the saved image that you create in the first place will make it less likely that changing the save image adds enough content to push XML over a 4096-byte boundary. Always good advice, when trying something new and potentially dangerous :) -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

The 15/03/13, Eric Blake wrote: <...> Step3: virsh save && virsh destroy. Step 5: virsh restore. Looks like a very good optimization for recent enough libvirt. Thanks! -- Nicolas Sebrecht

On 03/18/2013 05:39 AM, Nicolas Sebrecht wrote: Correct. Indeed, sounds like room for a future addition to libvirt, if qemu 1.5 gives us the additional hooks that I'd like to have. In particular, there is talk on the upstream qemu list about adding a blockbackup operation that would make capturing memory state and starting a backup job easier to manage than the current approach of starting a mirroring job, then capturing memory state and ending the mirroring job. A live-migration solution would have less downtime, but potentially take up more disk space. Again, there is talk on the upstream qemu list about how to optimize the amount of disk space taken by a live migration to a seekable file. Care to write a patch? The virsh man page is generated from libvirt.git:tools/virsh.pod. Again, if libvirt can be improved to do better external snapshot management, a sync of the xml/memory state file should be part of this effort. Yes, but that's true of any backup operation - you are committing to the disk space for both the live and the backup, in some form or another. Also, disk mirroring to a remote point is fairly easy to set up, by using nbd:... instead of /path/to/file as the destination for the blockcopy, and having an NBD server listening on the remote destination side. Probably a question better asked on the qemu list, but yes, my understanding is that if you have a disk mirror or backup job set up, then qemu has to manage to flush I/O to two locations instead of one, and might end up slightly slowing the guest as a result. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

The 13/03/13, Eric Blake wrote: I'm coming back to say that we now have a working solution in production based on external snapshots. It's written in python to execute to process detailed in this thread. The downtime is a few seconds. Thanks again. -- Nicolas Sebrecht