[libvirt] Problems live migrating a Windows VM - Inconsistent?
Hello, I noticed some issues migrating a VM 1) I had to have ssh key authentication exchanged or it would refuse to connect tunneled to the remote libvirtd server 2) When it attempts to finish live migrating the VM it fails saying it can't find the domain on the remote libvirtd server host However, if I run the command again to migrate, it DOES migrate the VM over, however the Windows guest is not usable, cant type/use mouse, but i can see the VM display fine. I can gracefully shut the VM down from virt-manager and I see the VM shut down gracefully from display attached via spice. This seems like two bugs? Versions: host kernel: 3.6.0-0.rc7.git0.1.fc18.x86_64 libvirt-XXXXXXX-0.10.2-3.fc18.x86_64 qemu-kvm-1.2.0-8.fc18.x86_64 (will upgrade to latest on koji but unrelated I believe) Command: LIBVIRT_DEBUG=1 virsh --debug=4 migrate --verbose --live --tunnelled --p2p Windows_7 qemu+ssh://root@172.18.2.3/system Output: ... <snip> 2012-09-30 17:59:14.814+0000: 24710: debug : virKeepAliveCheckMessage:384 : ka=0x266a470, client=0x2669ee0, msg=0x2669f48 2012-09-30 17:59:14.814+0000: 24710: debug : virNetClientIOEventLoopRemoveDone:1380 : Waking up sleep 0x2669cb0 2012-09-30 17:59:14.814+0000: 24708: debug : virNetClientIO:1754 : Woken up from sleep head=0x7f4830400e60 call=0x2669cb0 2012-09-30 17:59:14.814+0000: 24708: debug : virNetClientIO:1807 : All done with our call head=0x7f4830400e60 call=0x2669cb0 rv=0 2012-09-30 17:59:14.814+0000: 24708: debug : virNetMessageFree:73 : msg=0x2669640 nfds=0 cb=(nil) Migration: [ 95 %]2012-09-30 17:59:15.315+0000: 24708: debug : virDomainGetJobInfo:17129 : dom=0x2669e70, (VM: name=Windows, uuid=41562d70-d0a8-25bc-10ec-a3dc46b3d258), info=0x7fffb64d72d0 2012-09-30 17:59:15.315+0000: 24708: debug : virNetMessageNew:45 : msg=0x2669640 tracked=0 2012-09-30 17:59:15.315+0000: 24708: debug : virNetMessageEncodePayload:358 : Encode length as 64 2012-09-30 17:59:15.315+0000: 24708: debug : virNetClientSendInternal:1955 : RPC_CLIENT_MSG_TX_QUEUE: client=0x2669ee0 len=64 prog=536903814 vers=1 proc=163 type=0 status=0 serial=50 2012-09-30 17:59:15.315+0000: 24708: debug : virNetClientCallNew:1908 : New call 0x2669150: msg=0x2669640, expectReply=1, nonBlock=0 2012-09-30 17:59:15.315+0000: 24708: debug : virNetClientIO:1718 : Outgoing message prog=536903814 version=1 serial=50 proc=163 type=0 length=64 dispatch=0x7f4830400e60 2012-09-30 17:59:15.315+0000: 24708: debug : virNetClientIO:1744 : Going to sleep head=0x7f4830400e60 call=0x2669150 2012-09-30 17:59:15.315+0000: 24710: debug : virNetClientIOEventLoop:1558 : Woken up from poll by other thread 2012-09-30 17:59:15.317+0000: 24710: debug : virNetMessageDecodeLength:152 : Got length, now need 120 total (116 more) 2012-09-30 17:59:15.317+0000: 24710: debug : virNetClientCallDispatch:1127 : RPC_CLIENT_MSG_RX: client=0x2669ee0 len=120 prog=536903814 vers=1 proc=163 type=1 status=0 serial=50 2012-09-30 17:59:15.317+0000: 24710: debug : virKeepAliveCheckMessage:384 : ka=0x266a470, client=0x2669ee0, msg=0x2669f48 2012-09-30 17:59:15.317+0000: 24710: debug : virNetClientIOEventLoopRemoveDone:1380 : Waking up sleep 0x2669150 2012-09-30 17:59:15.317+0000: 24708: debug : virNetClientIO:1754 : Woken up from sleep head=0x7f4830400e60 call=0x2669150 2012-09-30 17:59:15.317+0000: 24708: debug : virNetClientIO:1807 : All done with our call head=0x7f4830400e60 call=0x2669150 rv=0 2012-09-30 17:59:15.317+0000: 24708: debug : virNetMessageFree:73 : msg=0x2669640 nfds=0 cb=(nil) Migration: [ 95 %]2012-09-30 17:59:15.751+0000: 24710: debug : virNetMessageDecodeLength:152 : Got length, now need 204 total (200 more) 2012-09-30 17:59:15.751+0000: 24710: debug : virNetClientCallDispatch:1127 : RPC_CLIENT_MSG_RX: client=0x2669ee0 len=204 prog=536903814 vers=1 proc=216 type=1 status=1 serial=8 2012-09-30 17:59:15.751+0000: 24710: debug : virKeepAliveCheckMessage:384 : ka=0x266a470, client=0x2669ee0, msg=0x2669f48 2012-09-30 17:59:15.751+0000: 24710: debug : virNetClientIOEventLoopPassTheBuck:1424 : Giving up the buck 0x7f4830400e60 2012-09-30 17:59:15.751+0000: 24710: debug : virNetClientIOEventLoopPassTheBuck:1438 : No thread to pass the buck to 2012-09-30 17:59:15.751+0000: 24710: debug : virEventPollUpdateHandle:146 : EVENT_POLL_UPDATE_HANDLE: watch=2 events=1 2012-09-30 17:59:15.751+0000: 24710: debug : virEventPollInterruptLocked:701 : Interrupting 2012-09-30 17:59:15.751+0000: 24710: debug : virNetClientIO:1807 : All done with our call head=(nil) call=0x7f4830400e60 rv=0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollRunOnce:625 : Poll got 1 event(s) 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollDispatchTimeouts:410 : Dispatch 0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollDispatchHandles:455 : Dispatch 1 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollDispatchHandles:469 : i=0 w=1 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollDispatchHandles:483 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupTimeouts:501 : Cleanup 0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupTimeouts:537 : Found 0 out of 0 timeout slots used, releasing 0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupHandles:549 : Cleanup 2 2012-09-30 17:59:15.751+0000: 24709: debug : virEventRunDefaultImpl:244 : running default event implementation 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupTimeouts:501 : Cleanup 0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupTimeouts:537 : Found 0 out of 0 timeout slots used, releasing 0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupHandles:549 : Cleanup 2 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollMakePollFDs:378 : Prepare n=0 w=1, f=4 e=1 d=0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollMakePollFDs:378 : Prepare n=1 w=2, f=6 e=1 d=0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCalculateTimeout:320 : Calculate expiry of 0 timers 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCalculateTimeout:346 : Timeout at 0 due in -1 ms 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollRunOnce:614 : EVENT_POLL_RUN: nhandles=2 timeout=-1 2012-09-30 17:59:15.751+0000: 24710: debug : virNetMessageFree:73 : msg=0x7f4830000de0 nfds=0 cb=(nil) 2012-09-30 17:59:15.751+0000: 24710: debug : virDomainFree:2281 : dom=0x7f4830000af0, (VM: name=Windows, uuid=41562d70-d0a8-25bc-10ec-a3dc46b3d258) 2012-09-30 17:59:15.751+0000: 24710: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x7f4830000af0 2012-09-30 17:59:15.751+0000: 24710: debug : virObjectUnref:137 : OBJECT_DISPOSE: obj=0x7f4830000af0 2012-09-30 17:59:15.751+0000: 24710: debug : virDomainDispose:225 : release domain 0x7f4830000af0 Windows 41562d70-d0a8-25bc-10ec-a3dc46b3d258 2012-09-30 17:59:15.751+0000: 24710: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x26693b0 2012-09-30 17:59:15.751+0000: 24708: debug : virDomainFree:2281 : dom=0x2669e70, (VM: name=Windows, uuid=41562d70-d0a8-25bc-10ec-a3dc46b3d258) 2012-09-30 17:59:15.751+0000: 24708: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x2669e70 2012-09-30 17:59:15.751+0000: 24708: debug : virObjectUnref:137 : OBJECT_DISPOSE: obj=0x2669e70 2012-09-30 17:59:15.751+0000: 24708: debug : virDomainDispose:225 : release domain 0x2669e70 Windows 41562d70-d0a8-25bc-10ec-a3dc46b3d258 2012-09-30 17:59:15.752+0000: 24708: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x26693b0 2012-09-30 17:59:15.752+0000: 24708: debug : virFileClose:72 : Closed fd 9 2012-09-30 17:59:15.752+0000: 24708: debug : virFileClose:72 : Closed fd 10 error: Domain not found: no domain with matching name 'Windows'
On Sunday, September 30, 2012 02:20:37 PM you wrote:
Hello,
I noticed some issues migrating a VM
1) I had to have ssh key authentication exchanged or it would refuse to connect tunneled to the remote libvirtd server
2) When it attempts to finish live migrating the VM it fails saying it can't find the domain on the remote libvirtd server host
However, if I run the command again to migrate, it DOES migrate the VM over, however the Windows guest is not usable, cant type/use mouse, but i can see the VM display fine. I can gracefully shut the VM down from virt-manager and I see the VM shut down gracefully from display attached via spice.
Correction, I can use the keyboard but not mouse, the mouse is attached as a USB device.
This seems like two bugs?
Versions:
host kernel: 3.6.0-0.rc7.git0.1.fc18.x86_64 libvirt-XXXXXXX-0.10.2-3.fc18.x86_64 qemu-kvm-1.2.0-8.fc18.x86_64 (will upgrade to latest on koji but unrelated I believe)
Command:
LIBVIRT_DEBUG=1 virsh --debug=4 migrate --verbose --live --tunnelled --p2p Windows_7 qemu+ssh://root@172.18.2.3/system
Output:
... <snip>
Output:
... <snip>
2012-09-30 17:59:14.814+0000: 24710: debug : virKeepAliveCheckMessage:384 : ka=0x266a470, client=0x2669ee0, msg=0x2669f48 2012-09-30 17:59:14.814+0000: 24710: debug : virNetClientIOEventLoopRemoveDone:1380 : Waking up sleep 0x2669cb0 2012-09-30 17:59:14.814+0000: 24708: debug : virNetClientIO:1754 : Woken up from sleep head=0x7f4830400e60 call=0x2669cb0 2012-09-30 17:59:14.814+0000: 24708: debug : virNetClientIO:1807 : All done with our call head=0x7f4830400e60 call=0x2669cb0 rv=0 2012-09-30 17:59:14.814+0000: 24708: debug : virNetMessageFree:73 : msg=0x2669640 nfds=0 cb=(nil) Migration: [ 95 %]2012-09-30 17:59:15.315+0000: 24708: debug : virDomainGetJobInfo:17129 : dom=0x2669e70, (VM: name=Windows, uuid=41562d70-d0a8-25bc-10ec-a3dc46b3d258), info=0x7fffb64d72d0 2012-09-30 17:59:15.315+0000: 24708: debug : virNetMessageNew:45 : msg=0x2669640 tracked=0 2012-09-30 17:59:15.315+0000: 24708: debug : virNetMessageEncodePayload:358 : Encode length as 64 2012-09-30 17:59:15.315+0000: 24708: debug : virNetClientSendInternal:1955 : RPC_CLIENT_MSG_TX_QUEUE: client=0x2669ee0 len=64 prog=536903814 vers=1 proc=163 type=0 status=0 serial=50 2012-09-30 17:59:15.315+0000: 24708: debug : virNetClientCallNew:1908 : New call 0x2669150: msg=0x2669640, expectReply=1, nonBlock=0 2012-09-30 17:59:15.315+0000: 24708: debug : virNetClientIO:1718 : Outgoing message prog=536903814 version=1 serial=50 proc=163 type=0 length=64 dispatch=0x7f4830400e60 2012-09-30 17:59:15.315+0000: 24708: debug : virNetClientIO:1744 : Going to sleep head=0x7f4830400e60 call=0x2669150 2012-09-30 17:59:15.315+0000: 24710: debug : virNetClientIOEventLoop:1558 : Woken up from poll by other thread 2012-09-30 17:59:15.317+0000: 24710: debug : virNetMessageDecodeLength:152 : Got length, now need 120 total (116 more) 2012-09-30 17:59:15.317+0000: 24710: debug : virNetClientCallDispatch:1127 : RPC_CLIENT_MSG_RX: client=0x2669ee0 len=120 prog=536903814 vers=1 proc=163 type=1 status=0 serial=50 2012-09-30 17:59:15.317+0000: 24710: debug : virKeepAliveCheckMessage:384 : ka=0x266a470, client=0x2669ee0, msg=0x2669f48 2012-09-30 17:59:15.317+0000: 24710: debug : virNetClientIOEventLoopRemoveDone:1380 : Waking up sleep 0x2669150 2012-09-30 17:59:15.317+0000: 24708: debug : virNetClientIO:1754 : Woken up from sleep head=0x7f4830400e60 call=0x2669150 012-09-30 17:59:15.317+0000: 24708: debug : virNetClientIO:1807 : All done with our call head=0x7f4830400e60 call=0x2669150 rv=0 2012-09-30 17:59:15.317+0000: 24708: debug : virNetMessageFree:73 : msg=0x2669640 nfds=0 cb=(nil) Migration: [ 95 %]2012-09-30 17:59:15.751+0000: 24710: debug : virNetMessageDecodeLength:152 : Got length, now need 204 total (200 more) 2012-09-30 17:59:15.751+0000: 24710: debug : virNetClientCallDispatch:1127 : RPC_CLIENT_MSG_RX: client=0x2669ee0 len=204 prog=536903814 vers=1 proc=216 type=1 status=1 serial=8 2012-09-30 17:59:15.751+0000: 24710: debug : virKeepAliveCheckMessage:384 : ka=0x266a470, client=0x2669ee0, msg=0x2669f48 2012-09-30 17:59:15.751+0000: 24710: debug : virNetClientIOEventLoopPassTheBuck:1424 : Giving up the buck 0x7f4830400e60 2012-09-30 17:59:15.751+0000: 24710: debug : virNetClientIOEventLoopPassTheBuck:1438 : No thread to pass the buck to 2012-09-30 17:59:15.751+0000: 24710: debug : virEventPollUpdateHandle:146 : EVENT_POLL_UPDATE_HANDLE: watch=2 events=1 2012-09-30 17:59:15.751+0000: 24710: debug : virEventPollInterruptLocked:701 : Interrupting 2012-09-30 17:59:15.751+0000: 24710: debug : virNetClientIO:1807 : All done with our call head=(nil) call=0x7f4830400e60 rv=0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollRunOnce:625 : Poll got 1 event(s) 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollDispatchTimeouts:410 : Dispatch 0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollDispatchHandles:455 : Dispatch 1 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollDispatchHandles:469 : i=0 w=1 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollDispatchHandles:483 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupTimeouts:501 : Cleanup 0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupTimeouts:537 : Found 0 out of 0 timeout slots used, releasing 0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupHandles:549 : Cleanup 2 2012-09-30 17:59:15.751+0000: 24709: debug : virEventRunDefaultImpl:244 : running default event implementation 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupTimeouts:501 : Cleanup 0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupTimeouts:537 : Found 0 out of 0 timeout slots used, releasing 0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCleanupHandles:549 : Cleanup 2 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollMakePollFDs:378 : Prepare n=0 w=1, f=4 e=1 d=0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollMakePollFDs:378 : Prepare n=1 w=2, f=6 e=1 d=0 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCalculateTimeout:320 : Calculate expiry of 0 timers 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollCalculateTimeout:346 : Timeout at 0 due in -1 ms 2012-09-30 17:59:15.751+0000: 24709: debug : virEventPollRunOnce:614 : EVENT_POLL_RUN: nhandles=2 timeout=-1 2012-09-30 17:59:15.751+0000: 24710: debug : virNetMessageFree:73 : msg=0x7f4830000de0 nfds=0 cb=(nil) 2012-09-30 17:59:15.751+0000: 24710: debug : virDomainFree:2281 : dom=0x7f4830000af0, (VM: name=Windows, uuid=41562d70-d0a8-25bc-10ec-a3dc46b3d258) 2012-09-30 17:59:15.751+0000: 24710: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x7f4830000af0 2012-09-30 17:59:15.751+0000: 24710: debug : virObjectUnref:137 : OBJECT_DISPOSE: obj=0x7f4830000af0 2012-09-30 17:59:15.751+0000: 24710: debug : virDomainDispose:225 : release domain 0x7f4830000af0 Windows 41562d70-d0a8-25bc-10ec-a3dc46b3d258 2012-09-30 17:59:15.751+0000: 24710: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x26693b0 2012-09-30 17:59:15.751+0000: 24708: debug : virDomainFree:2281 : dom=0x2669e70, (VM: name=Windows, uuid=41562d70-d0a8-25bc-10ec-a3dc46b3d258) 2012-09-30 17:59:15.751+0000: 24708: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x2669e70 2012-09-30 17:59:15.751+0000: 24708: debug : virObjectUnref:137 : OBJECT_DISPOSE: obj=0x2669e70 2012-09-30 17:59:15.751+0000: 24708: debug : virDomainDispose:225 : release domain 0x2669e70 Windows 41562d70-d0a8-25bc-10ec-a3dc46b3d258 2012-09-30 17:59:15.752+0000: 24708: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x26693b0 2012-09-30 17:59:15.752+0000: 24708: debug : virFileClose:72 : Closed fd 9 2012-09-30 17:59:15.752+0000: 24708: debug : virFileClose:72 : Closed fd 10 error: Domain not found: no domain with matching name 'Windows'
On 09/30/2012 12:20 PM, Shawn Starr wrote:
Hello,
I noticed some issues migrating a VM
1) I had to have ssh key authentication exchanged or it would refuse to connect tunneled to the remote libvirtd server
Correct - per http://libvirt.org/migration.html, under tunnelled migration, there is this warning: The destination URI must be reachable using the source libvirtd credentials (which are not necessarily the same as the credentials of the client in connecting to the source). There has also been a recent addition of a new qemu+libssh2://host/system URI, with better credential handling through libvirt APIs, although I'm not sure if the new URI is capable of handling the authentication exchange as part of the libvirt migrate command.
2) When it attempts to finish live migrating the VM it fails saying it can't find the domain on the remote libvirtd server host
Can you print the exact error message it gave? [Oh, I see you did that below]
However, if I run the command again to migrate, it DOES migrate the VM over, however the Windows guest is not usable, cant type/use mouse, but i can see the VM display fine. I can gracefully shut the VM down from virt-manager and I see the VM shut down gracefully from display attached via spice.
As for the mouse not being usable after migration, that sounds more like an issue in qemu; you might get better results asking on a qemu list, and/or upgrading to newer qemu on both source and destination machines before attempting another migration.
LIBVIRT_DEBUG=1 virsh --debug=4 migrate --verbose --live --tunnelled --p2p Windows_7 qemu+ssh://root@172.18.2.3/system
Output:
2012-09-30 17:59:15.751+0000: 24710: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x26693b0 2012-09-30 17:59:15.751+0000: 24708: debug : virDomainFree:2281 : dom=0x2669e70, (VM: name=Windows, uuid=41562d70-d0a8-25bc-10ec-a3dc46b3d258) 2012-09-30 17:59:15.751+0000: 24708: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x2669e70 2012-09-30 17:59:15.751+0000: 24708: debug : virObjectUnref:137 : OBJECT_DISPOSE: obj=0x2669e70 2012-09-30 17:59:15.751+0000: 24708: debug : virDomainDispose:225 : release domain 0x2669e70 Windows 41562d70-d0a8-25bc-10ec-a3dc46b3d258 2012-09-30 17:59:15.752+0000: 24708: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x26693b0 2012-09-30 17:59:15.752+0000: 24708: debug : virFileClose:72 : Closed fd 9 2012-09-30 17:59:15.752+0000: 24708: debug : virFileClose:72 : Closed fd 10 error: Domain not found: no domain with matching name 'Windows'
It sounds like this particular migration attempt hit a case where qemu failed to migrate, so libvirt aborted the migration (perhaps with a confusing message, though). -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
participants (2)
-
Eric Blake -
Shawn Starr