[libvirt] qemu: managedsave vs. save
Hi! My KVM hosts share the same filesystem and I'm facing an issue using managedsave. If I save vmX using managedsave on hostA and restore it later using "virsh restore" in hostB the qemu process consumes 100% CPU and makes no progress. On the other hand, if I save vmX using save the restore works fine on hostB. Why can't I restore vms saved by managedsave using restore? Or is this a known issue on libvirt-0.10.X (CentOS 6)? -- Thanks, //richard
On Fri, May 02, 2014 at 02:08:28PM +0200, Richard Weinberger wrote:
Hi!
My KVM hosts share the same filesystem and I'm facing an issue using managedsave. If I save vmX using managedsave on hostA and restore it later using "virsh restore" in hostB the qemu process consumes 100% CPU and makes no progress. On the other hand, if I save vmX using save the restore works fine on hostB.
FWIW, you use 'managedsave' then you shouldn't use 'restore' - you should just 'start' the guest as normal and libvirt will automagically use the managed save image. Of course this doesn't explain the problem you see - the result of managedsave should be identical to the result of save. They use the same QEMU code internally, the only difference being that in one case you decide the filename and in the other case libvirt decides the filename Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Fri, May 2, 2014 at 2:16 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, May 02, 2014 at 02:08:28PM +0200, Richard Weinberger wrote:
Hi!
My KVM hosts share the same filesystem and I'm facing an issue using managedsave. If I save vmX using managedsave on hostA and restore it later using "virsh restore" in hostB the qemu process consumes 100% CPU and makes no progress. On the other hand, if I save vmX using save the restore works fine on hostB.
FWIW, you use 'managedsave' then you shouldn't use 'restore' - you should just 'start' the guest as normal and libvirt will automagically use the managed save image.
I cannot use start because on hostB vmX is not defined. :-(
Of course this doesn't explain the problem you see - the result of managedsave should be identical to the result of save. They use the same QEMU code internally, the only difference being that in one case you decide the filename and in the other case libvirt decides the filename
Hmm, upon restore of a managesave'd vm libvirtd logs this: 2014-05-02 11:40:09.423+0000: 15974: error : qemuMonitorIO:614 : Interner Fehler End of file from monitor Maybe it gives you a hint. -- Thanks, //richard
On Fri, May 02, 2014 at 02:21:08PM +0200, Richard Weinberger wrote:
On Fri, May 2, 2014 at 2:16 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, May 02, 2014 at 02:08:28PM +0200, Richard Weinberger wrote:
Hi!
My KVM hosts share the same filesystem and I'm facing an issue using managedsave. If I save vmX using managedsave on hostA and restore it later using "virsh restore" in hostB the qemu process consumes 100% CPU and makes no progress. On the other hand, if I save vmX using save the restore works fine on hostB.
FWIW, you use 'managedsave' then you shouldn't use 'restore' - you should just 'start' the guest as normal and libvirt will automagically use the managed save image.
I cannot use start because on hostB vmX is not defined. :-(
Oh, I missed that you were moving VMs from one host to another.
Of course this doesn't explain the problem you see - the result of managedsave should be identical to the result of save. They use the same QEMU code internally, the only difference being that in one case you decide the filename and in the other case libvirt decides the filename
Hmm, upon restore of a managesave'd vm libvirtd logs this: 2014-05-02 11:40:09.423+0000: 15974: error : qemuMonitorIO:614 : Interner Fehler End of file from monitor
Maybe it gives you a hint.
Indicates that QEMU quit - what is in /var/log/libvirt/qemu/$GUESTNAME.log Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Fri, May 2, 2014 at 2:26 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, May 02, 2014 at 02:21:08PM +0200, Richard Weinberger wrote:
On Fri, May 2, 2014 at 2:16 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, May 02, 2014 at 02:08:28PM +0200, Richard Weinberger wrote:
Hi!
My KVM hosts share the same filesystem and I'm facing an issue using managedsave. If I save vmX using managedsave on hostA and restore it later using "virsh restore" in hostB the qemu process consumes 100% CPU and makes no progress. On the other hand, if I save vmX using save the restore works fine on hostB.
FWIW, you use 'managedsave' then you shouldn't use 'restore' - you should just 'start' the guest as normal and libvirt will automagically use the managed save image.
I cannot use start because on hostB vmX is not defined. :-(
Oh, I missed that you were moving VMs from one host to another.
Yep. I can also not define them on hostB because the .save files are not located in /var/lib/libvirt/qemu/save and therefore the start command would not find them.
Of course this doesn't explain the problem you see - the result of managedsave should be identical to the result of save. They use the same QEMU code internally, the only difference being that in one case you decide the filename and in the other case libvirt decides the filename
Hmm, upon restore of a managesave'd vm libvirtd logs this: 2014-05-02 11:40:09.423+0000: 15974: error : qemuMonitorIO:614 : Interner Fehler End of file from monitor
Maybe it gives you a hint.
Indicates that QEMU quit - what is in /var/log/libvirt/qemu/$GUESTNAME.log
Nothing special: 2014-05-02 11:38:19.544+0000: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name 2ndvm,process=qemu:2ndvm -S -M rhel6.1.0 -enable-kvm -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 735b6912-3cf4-4c68-9780-c6d979c88ac8 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/2ndvm.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/srv/galadriel_vms1/vmdisks/2ndvm/hd0.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:0a:00:01:1c,bus=pci.0,addr=0x3 -vnc 127.0.0.1:1 -k de -vga cirrus -incoming fd:21 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 qemu: terminating on signal 15 from pid 15604 The SIGTERM was sent by me after a few minuues... -- Thanks, //richard
On Fri, May 2, 2014 at 2:16 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, May 02, 2014 at 02:08:28PM +0200, Richard Weinberger wrote:
Hi!
My KVM hosts share the same filesystem and I'm facing an issue using managedsave. If I save vmX using managedsave on hostA and restore it later using "virsh restore" in hostB the qemu process consumes 100% CPU and makes no progress. On the other hand, if I save vmX using save the restore works fine on hostB.
FWIW, you use 'managedsave' then you shouldn't use 'restore' - you should just 'start' the guest as normal and libvirt will automagically use the managed save image.
Of course this doesn't explain the problem you see - the result of managedsave should be identical to the result of save. They use the same QEMU code internally, the only difference being that in one case you decide the filename and in the other case libvirt decides the filename
BTW: On a different system with libvirtd 1.2.3 restore works fine from managedsave'd domains. Looks like I have to upgrade my libvirt. :-\ -- Thanks, //richard
On 05/02/2014 03:38 PM, Richard Weinberger wrote:
On Fri, May 2, 2014 at 2:16 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, May 02, 2014 at 02:08:28PM +0200, Richard Weinberger wrote:
Hi!
My KVM hosts share the same filesystem and I'm facing an issue using managedsave. If I save vmX using managedsave on hostA and restore it later using "virsh restore" in hostB the qemu process consumes 100% CPU and makes no progress. On the other hand, if I save vmX using save the restore works fine on hostB. FWIW, you use 'managedsave' then you shouldn't use 'restore' - you should just 'start' the guest as normal and libvirt will automagically use the managed save image.
Of course this doesn't explain the problem you see - the result of managedsave should be identical to the result of save. They use the same QEMU code internally, the only difference being that in one case you decide the filename and in the other case libvirt decides the filename BTW: On a different system with libvirtd 1.2.3 restore works fine from managedsave'd domains.
Looks like I have to upgrade my libvirt. :-\
But you still should not use a combination of managedsave + [manually moving the config & save file to other system] + restore to migrate a guest from one host to another. This is misuse of managedsave, and extra work that you shouldn't have to do. (managedsave is really intended to be used to temporarily pause a guest so that, e.g., the host can be rebooted, after which the guest can be restarted *on the same host* by using virsh start; this is used by the libvirt-guests service. Although starting this saved image on a different host may currently work, there is no guarantee that it will continue to work in the future). Instead look into the virsh migrate command. It will do all the dirty work for you, including removing the guest definition from the source host and defining it on the destination host if you want, and it minimizes guest downtime to "something very short". It can also migrate the storage if you need it to (although this increases the migration time considerably). (BTW, both virsh save and virsh managedsave are implemented by telling qemu to "migrate to disk", so in the end it's all the same thing; all that changes is the level of details that libvirt manages for you)
On Fri, May 2, 2014 at 3:43 PM, Laine Stump <laine@laine.org> wrote:
On 05/02/2014 03:38 PM, Richard Weinberger wrote:
On Fri, May 2, 2014 at 2:16 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Fri, May 02, 2014 at 02:08:28PM +0200, Richard Weinberger wrote:
Hi!
My KVM hosts share the same filesystem and I'm facing an issue using managedsave. If I save vmX using managedsave on hostA and restore it later using "virsh restore" in hostB the qemu process consumes 100% CPU and makes no progress. On the other hand, if I save vmX using save the restore works fine on hostB. FWIW, you use 'managedsave' then you shouldn't use 'restore' - you should just 'start' the guest as normal and libvirt will automagically use the managed save image.
Of course this doesn't explain the problem you see - the result of managedsave should be identical to the result of save. They use the same QEMU code internally, the only difference being that in one case you decide the filename and in the other case libvirt decides the filename BTW: On a different system with libvirtd 1.2.3 restore works fine from managedsave'd domains.
Looks like I have to upgrade my libvirt. :-\
But you still should not use a combination of managedsave + [manually moving the config & save file to other system] + restore to migrate a guest from one host to another. This is misuse of managedsave, and extra work that you shouldn't have to do. (managedsave is really intended to be used to temporarily pause a guest so that, e.g., the host can be rebooted, after which the guest can be restarted *on the same host* by using virsh start; this is used by the libvirt-guests service. Although starting this saved image on a different host may currently work, there is no guarantee that it will continue to work in the future).
Then please clearly state this in the man page and create .save images which is unable to be resumed by "restore" to prevent further misuse. :)
Instead look into the virsh migrate command. It will do all the dirty work for you, including removing the guest definition from the source host and defining it on the destination host if you want, and it minimizes guest downtime to "something very short". It can also migrate the storage if you need it to (although this increases the migration time considerably).
I cannot use migrate because I'm using DRBD. Takeover from hostA to hostB works as follows: 1. save all VMs do disk 2. unmount the fs on hostA 3. make hostA secondary in the DBRD cluster 4. make hostB primary 5. mount the fs on hostB 6. restore saved VMs on hostB Looks like I have to update /etc/init.d/libvirt-guests to use a plain "save" instead of "managedsave". -- Thanks, //richard
On 05/02/2014 04:52 PM, Richard Weinberger wrote: > On Fri, May 2, 2014 at 3:43 PM, Laine Stump <laine@laine.org> wrote: >> On 05/02/2014 03:38 PM, Richard Weinberger wrote: >>> On Fri, May 2, 2014 at 2:16 PM, Daniel P. Berrange <berrange@redhat.com> wrote: >>>> On Fri, May 02, 2014 at 02:08:28PM +0200, Richard Weinberger wrote: >>>>> Hi! >>>>> >>>>> My KVM hosts share the same filesystem and I'm facing an issue using >>>>> managedsave. >>>>> If I save vmX using managedsave on hostA and restore it later using >>>>> "virsh restore" in hostB >>>>> the qemu process consumes 100% CPU and makes no progress. >>>>> On the other hand, if I save vmX using save the restore works fine on hostB. >>>> FWIW, you use 'managedsave' then you shouldn't use 'restore' - you >>>> should just 'start' the guest as normal and libvirt will automagically >>>> use the managed save image. >>>> >>>> Of course this doesn't explain the problem you see - the result of >>>> managedsave should be identical to the result of save. They use the >>>> same QEMU code internally, the only difference being that in one >>>> case you decide the filename and in the other case libvirt decides >>>> the filename >>> BTW: On a different system with libvirtd 1.2.3 restore works fine >>> from managedsave'd domains. >>> >>> Looks like I have to upgrade my libvirt. :-\ >> But you still should not use a combination of managedsave + [manually >> moving the config & save file to other system] + restore to migrate a >> guest from one host to another. This is misuse of managedsave, and extra >> work that you shouldn't have to do. (managedsave is really intended to >> be used to temporarily pause a guest so that, e.g., the host can be >> rebooted, after which the guest can be restarted *on the same host* by >> using virsh start; this is used by the libvirt-guests service. Although >> starting this saved image on a different host may currently work, there >> is no guarantee that it will continue to work in the future). > Then please clearly state this in the man page and create .save images > which is unable to be resumed by "restore" to prevent further misuse. :) > >> Instead look into the virsh migrate command. It will do all the dirty >> work for you, including removing the guest definition from the source >> host and defining it on the destination host if you want, and it >> minimizes guest downtime to "something very short". It can also migrate >> the storage if you need it to (although this increases the migration >> time considerably). > I cannot use migrate because I'm using DRBD. > Takeover from hostA to hostB works as follows: > 1. save all VMs do disk > 2. unmount the fs on hostA > 3. make hostA secondary in the DBRD cluster > 4. make hostB primary > 5. mount the fs on hostB > 6. restore saved VMs on hostB Ah, so the two machines are not simultaneously available. That changes a lot. > Looks like I have to update /etc/init.d/libvirt-guests to use a plain "save" > instead of "managedsave". Well, as Daniel said earlier, there is no difference in produced file between save and managedsave, and I don't know that either of these is explicitly documented as creating a saved image that can be restarted on a different machine. This is what led to my comment about misusing managedsave. But your case sounds like a "legitimate misuse" :-) (since migration requires that both machines be available at the same time). So lacking a more formally approved method, there shouldn't be any problem with what you are doing, but keep in mind that if there is any difference between the software/hardware of the two machines that could prevent the saved image from being properly restored on the new host, you'll need to take care of the fallout of that yourself (if you were able to use migration, the migration would fail and leave you with the guest still running on the original host). (I wonder if we should officially state that the saved image can be safely restored on a different machine as long as it is "similar" (whatever that means) so that people in this situation can safely do as you're doing without fear of breakage after some hypothetical future change. Or maybe we *do* say that somewhere and I've just never seen it...)
On Fri, May 2, 2014 at 4:36 PM, Laine Stump <laine@laine.org> wrote: > On 05/02/2014 04:52 PM, Richard Weinberger wrote: >> On Fri, May 2, 2014 at 3:43 PM, Laine Stump <laine@laine.org> wrote: >>> On 05/02/2014 03:38 PM, Richard Weinberger wrote: >>>> On Fri, May 2, 2014 at 2:16 PM, Daniel P. Berrange <berrange@redhat.com> wrote: >>>>> On Fri, May 02, 2014 at 02:08:28PM +0200, Richard Weinberger wrote: >>>>>> Hi! >>>>>> >>>>>> My KVM hosts share the same filesystem and I'm facing an issue using >>>>>> managedsave. >>>>>> If I save vmX using managedsave on hostA and restore it later using >>>>>> "virsh restore" in hostB >>>>>> the qemu process consumes 100% CPU and makes no progress. >>>>>> On the other hand, if I save vmX using save the restore works fine on hostB. >>>>> FWIW, you use 'managedsave' then you shouldn't use 'restore' - you >>>>> should just 'start' the guest as normal and libvirt will automagically >>>>> use the managed save image. >>>>> >>>>> Of course this doesn't explain the problem you see - the result of >>>>> managedsave should be identical to the result of save. They use the >>>>> same QEMU code internally, the only difference being that in one >>>>> case you decide the filename and in the other case libvirt decides >>>>> the filename >>>> BTW: On a different system with libvirtd 1.2.3 restore works fine >>>> from managedsave'd domains. >>>> >>>> Looks like I have to upgrade my libvirt. :-\ >>> But you still should not use a combination of managedsave + [manually >>> moving the config & save file to other system] + restore to migrate a >>> guest from one host to another. This is misuse of managedsave, and extra >>> work that you shouldn't have to do. (managedsave is really intended to >>> be used to temporarily pause a guest so that, e.g., the host can be >>> rebooted, after which the guest can be restarted *on the same host* by >>> using virsh start; this is used by the libvirt-guests service. Although >>> starting this saved image on a different host may currently work, there >>> is no guarantee that it will continue to work in the future). >> Then please clearly state this in the man page and create .save images >> which is unable to be resumed by "restore" to prevent further misuse. :) >> >>> Instead look into the virsh migrate command. It will do all the dirty >>> work for you, including removing the guest definition from the source >>> host and defining it on the destination host if you want, and it >>> minimizes guest downtime to "something very short". It can also migrate >>> the storage if you need it to (although this increases the migration >>> time considerably). >> I cannot use migrate because I'm using DRBD. >> Takeover from hostA to hostB works as follows: >> 1. save all VMs do disk >> 2. unmount the fs on hostA >> 3. make hostA secondary in the DBRD cluster >> 4. make hostB primary >> 5. mount the fs on hostB >> 6. restore saved VMs on hostB > > Ah, so the two machines are not simultaneously available. That changes a > lot. They are simultaneously available, but only one can access the filesystem by the nature of DRBD. >> Looks like I have to update /etc/init.d/libvirt-guests to use a plain "save" >> instead of "managedsave". > > Well, as Daniel said earlier, there is no difference in produced file > between save and managedsave, and I don't know that either of these is > explicitly documented as creating a saved image that can be restarted on > a different machine. This is what led to my comment about misusing > managedsave. But your case sounds like a "legitimate misuse" :-) (since > migration requires that both machines be available at the same time). > > So lacking a more formally approved method, there shouldn't be any > problem with what you are doing, but keep in mind that if there is any > difference between the software/hardware of the two machines that could > prevent the saved image from being properly restored on the new host, > you'll need to take care of the fallout of that yourself (if you were > able to use migration, the migration would fail and leave you with the > guest still running on the original host). Both hosts are identical by design. > (I wonder if we should officially state that the saved image can be > safely restored on a different machine as long as it is "similar" > (whatever that means) so that people in this situation can safely do as > you're doing without fear of breakage after some hypothetical future > change. Or maybe we *do* say that somewhere and I've just never seen it...) BTW: managedsave && restore fails also on the same host. Only save && restore works fine. -- Thanks, //richard
participants (3)
-
Daniel P. Berrange -
Laine Stump -
Richard Weinberger