I'm facing a strange issue while doing a migration from an hypervisor to another one. The migration takes for ever to start moving the memory.

The VM had no workload what so ever, just a basic ubuntu image. The versions on the hypervisors are: libvirt 1.2.21, qemu 1.2.3

Command to launche the migration:

virsh migrate --verbose --live --abort-on-error --tunnelled --p2p --auto-converge --copy-storage-inc --xml vm-6160.xml 6160 qemu+tls://<destination_hypervisor>/system

Here is the log output, look at the time elapsed:

root@virt-hv009:~# virsh domjobinfo 6160

Job type: Unbounded

Time elapsed: 27518 ms

Data processed: 21.506 GiB

Data remaining: 29.003 GiB

Data total: 50.509 GiB

Memory processed: 0.000 B

Memory remaining: 520.820 MiB

Memory total: 520.820 MiB

File processed: 21.506 GiB

File remaining: 28.494 GiB

File total: 50.000 GiB

Constant pages: 0

Normal pages: 0

Normal data: 0.000 B

Expected downtime: 300 ms

Setup time: 6 ms

root@virt-hv009:~# virsh domjobinfo 6160

Job type: Unbounded

Time elapsed: 32331 ms

Data processed: 25.475 GiB

Data remaining: 25.034 GiB

Data total: 50.509 GiB

Memory processed: 0.000 B

Memory remaining: 520.820 MiB

Memory total: 520.820 MiB

File processed: 25.475 GiB

File remaining: 24.525 GiB

File total: 50.000 GiB

Constant pages: 0

Normal pages: 0

Normal data: 0.000 B

Expected downtime: 300 ms

Setup time: 6 ms

root@virt-hv009:~# virsh domjobinfo 6160

Job type: Unbounded

Time elapsed: 49543 ms

Data processed: 50.000 GiB

Data remaining: 520.820 MiB

Data total: 50.509 GiB

Memory processed: 0.000 B

Memory remaining: 520.820 MiB

Memory total: 520.820 MiB

File processed: 50.000 GiB

File remaining: 0.000 B

File total: 50.000 GiB

Constant pages: 0

Normal pages: 0

Normal data: 0.000 B

Expected downtime: 300 ms

Setup time: 6 ms

^^^^ Here after 49 sec, the disk has been process, but the memory hasn't started

I skip the logs, but below is the output 5minutes & 56 seconds later, still nothing:

root@virt-hv009:~# virsh domjobinfo 6160

Job type: Unbounded

Time elapsed: 356919 ms

Data processed: 50.000 GiB

Data remaining: 520.820 MiB

Data total: 50.509 GiB

Memory processed: 0.000 B

Memory remaining: 520.820 MiB

Memory total: 520.820 MiB

File processed: 50.000 GiB

File remaining: 0.000 B

File total: 50.000 GiB

Constant pages: 0

Normal pages: 0

Normal data: 0.000 B

Expected downtime: 300 ms

Setup time: 6 ms

Just after it started to move the memory, but 6 minutes later!

root@virt-hv009:~# virsh domjobinfo 6160

Job type: Unbounded

Time elapsed: 360092 ms

Data processed: 50.052 GiB

Data remaining: 453.895 MiB

Data total: 50.509 GiB

Memory processed: 53.224 MiB

Memory remaining: 453.895 MiB

Memory total: 520.820 MiB

Memory bandwidth: 98.172 MiB/s

File processed: 50.000 GiB

File remaining: 0.000 B

File total: 50.000 GiB

Constant pages: 3541

Normal pages: 13591

Normal data: 53.090 MiB

Expected downtime: 300 ms

Setup time: 6 ms

root@virt-hv009:~# virsh domjobinfo 6160

error: failed to get domain '6160'

error: Domain not found: no domain with matching name '6160'

Here the migration is done. But what made it wait more than 5 minutes to start moving the memory? Does anyone knows? Or have any idea what could produce this latency.