Re: [libvirt] [PATCH v4 0/9] Selective block device migration implementation

On Wed, Jun 17, 2015 at 17:31:03 +0200, Kashyap Chamarthy wrote: > On Tue, Jun 16, 2015 at 01:42:02AM +0300, Pavel Boldin wrote: ... > libvirtd debug log[1] from source (destination log is empty)): > > [. . .] > 2015-06-17 15:13:53.317+0000: 781: debug : virDomainMigratePerform3Params:5202 : dom=0x7f2118f13c40, (VM: name=cvm1, uuid=ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e), dconnuri=qemu+tcp://root@devstack3/system, params=0x7f2118f12a90, nparams=1, cookiein=(nil), cookieinlen=0, cookieout=0x7f2106f38ba8, cookieoutlen=0x7f2106f38ba4, flags=3 > 2015-06-17 15:13:53.317+0000: 781: debug : virDomainMigratePerform3Params:5203 : params["migrate_disks"]=(string)vdb > 2015-06-17 15:13:53.317+0000: 781: debug : qemuMigrationPerform:5238 : driver=0x7f20f416b840, conn=0x7f20dc005c30, vm=0x7f20f41e9640, xmlin=<null>, dconnuri=qemu+tcp://root@devstack3/system, uri=<null>, graphicsuri=<null>, listenAddress=<null>, nmigrate_disks=1, migrate_disks=0x7f2118f13930, cookiein=<null>, cookieinlen=0, cookieout=0x7f2106f38ba8, cookieoutlen=0x7f2106f38ba4, flags=3, dname=<null>, resource=0, v3proto=1 > 2015-06-17 15:13:53.317+0000: 781: debug : qemuDomainObjBeginJobInternal:1397 : Starting async job: none (async=migration out vm=0x7f20f41e9640 name=cvm1) > 2015-06-17 15:13:53.317+0000: 781: debug : qemuDomainObjBeginJobInternal:1414 : Waiting for async job (vm=0x7f20f41e9640 name=cvm1) > 2015-06-17 15:13:53.821+0000: 782: debug : virThreadJobSet:96 : Thread 782 (virNetServerHandleJob) is now running job remoteDispatchDomainGetJobInfo > 2015-06-17 15:13:53.821+0000: 782: debug : virDomainGetJobInfo:8808 : dom=0x7f20dc008c30, (VM: name=cvm1, uuid=ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e), info=0x7f2106737b50 > 2015-06-17 15:13:53.821+0000: 782: debug : virThreadJobClear:121 : Thread 782 (virNetServerHandleJob) finished job remoteDispatchDomainGetJobInfo with ret=0 > 2015-06-17 15:13:54.325+0000: 780: debug : virThreadJobSet:96 : Thread 780 (virNetServerHandleJob) is now running job remoteDispatchDomainGetJobInfo > 2015-06-17 15:13:54.325+0000: 780: debug : virDomainGetJobInfo:8808 : dom=0x7f20dc008c30, (VM: name=cvm1, uuid=ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e), info=0x7f2107739b50 > 2015-06-17 15:13:54.325+0000: 780: debug : virThreadJobClear:121 : Thread 780 (virNetServerHandleJob) finished job remoteDispatchDomainGetJobInfo with ret=0 > [. . .] > remoteDispatchDomainMigratePerform3Params, 784 remoteDispatchDomainMigratePerform3Params) for (520s, 520s) > 2015-06-17 15:14:23.320+0000: 781: error : qemuDomainObjBeginJobInternal:1492 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePerform3Params) > 2015-06-17 15:14:23.320+0000: 781: debug : virThreadJobClear:121 : Thread 781 (virNetServerHandleJob) finished job remoteDispatchDomainMigratePerform3Params with ret=-1 > 2015-06-17 15:14:23.320+0000: 783: debug : virThreadJobSet:96 : Thread 783 (virNetServerHandleJob) is now running job remoteDispatchConnectClose > 2015-06-17 15:14:23.320+0000: 783: debug : virThreadJobClear:121 : Thread 783 (virNetServerHandleJob) finished job remoteDispatchConnectClose with ret=0 > > > How can I mitigate this? (I realize this is not due to these patches, > proably something with my test environment.) > > Since this is non-shared storage migration, I tried to supply > '--copy-storage-inc' to no avail (same error as above). > > Probably I should test by building local RPMs. > > [1] https://kashyapc.fedorapeople.org/virt/temp/libvirtd-log-selective-blockd... Could you upload a complete log somewhere? It seems a previously started migration is waiting for a response from QEMU. Or alternatively, it failed to release the jobs. I'd like to see the logs from the previous migration attempt.