unable to migrate: virPortAllocatorSetUsed:299 : internal error: Failed to reserve port 49153
On libvirt 6.8.0 and qemu 5.1.0, when trying to live migrate "error: internal error: Failed to reserve port" error is received and migration does not succeed: virsh # migrate cartridge qemu+tls://ratchet.lan/system --live --persistent --undefinesource --copy-storage-all --verbose error: internal error: Failed to reserve port 49153 virsh # On target host with debug logs, nothing interesting but the error itself is found in the logs ... 2020-10-12 02:11:33.852+0000: 6871: debug : qemuMonitorJSONIOProcessLine:220 : Line [{"return": {}, "id": "libvirt-373"}] 2020-10-12 02:11:33.852+0000: 6871: info : qemuMonitorJSONIOProcessLine:239 : QEMU_MONITOR_RECV_REPLY: mon=0x7fe784255020 reply={"return": {}, "id": "libvirt-373"} 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjExitMonitorInternal:5615 : Exited monitor (mon=0x7fe784255020 vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjEndJob:1140 : Stopping job: async nested (async=migration in vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: error : virPortAllocatorSetUsed:299 : internal error: Failed to reserve port 49153 2020-10-12 02:11:33.852+0000: 6825: debug : qemuMigrationParamsReset:1206 : Resetting migration parameters 0x7fe784257c30, flags 0x59 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjBeginJobInternal:835 : Starting job: job=async nested agentJob=none asyncJob=none (vm=0x55f086c81ea0 name=cartridge, current job=none agentJob=none async=migration in) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjBeginJobInternal:887 : Started job: async nested (async=migration in vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjEnterMonitorInternal:5590 : Entering monitor (mon=0x7fe784255020 vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuMonitorSetMigrationCapabilities:3853 : mon:0x7fe784255020 vm:0x55f086c81ea0 fd:30 2020-10-12 02:11:33.852+0000: 6825: info : qemuMonitorSend:944 : QEMU_MONITOR_SEND_MSG: mon=0x7fe784255020 msg={"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability> fd=-1 ... Full logs: destination: https://drive.google.com/file/d/1g986SbSVijvwZd8d7xDrJwo_AmKH-JnV/view?usp=s... source: https://drive.google.com/file/d/1lsV2EOBxF7xH5-lgz2Psh9YSkOePXvAd/view?usp=s... On target host, nothing is listening on the target port: ratchet /var/log/libvirt/qemu # netstat -lnp | grep 49153 | wc -l 0 and nc -l 49153 succeeds without issues. Do you have any suggestions on how to proceed here?
This problem reliably reproduces for me in 6.8.0 and at HEAD but not in 6.7.0 with no change in environment or VM configs. Should I file a bug? On Sun, Oct 11, 2020 at 7:46 PM Vjaceslavs Klimovs <vklimovs@gmail.com> wrote:
On libvirt 6.8.0 and qemu 5.1.0, when trying to live migrate "error: internal error: Failed to reserve port" error is received and migration does not succeed:
virsh # migrate cartridge qemu+tls://ratchet.lan/system --live --persistent --undefinesource --copy-storage-all --verbose error: internal error: Failed to reserve port 49153
virsh #
On target host with debug logs, nothing interesting but the error itself is found in the logs
... 2020-10-12 02:11:33.852+0000: 6871: debug : qemuMonitorJSONIOProcessLine:220 : Line [{"return": {}, "id": "libvirt-373"}] 2020-10-12 02:11:33.852+0000: 6871: info : qemuMonitorJSONIOProcessLine:239 : QEMU_MONITOR_RECV_REPLY: mon=0x7fe784255020 reply={"return": {}, "id": "libvirt-373"} 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjExitMonitorInternal:5615 : Exited monitor (mon=0x7fe784255020 vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjEndJob:1140 : Stopping job: async nested (async=migration in vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: error : virPortAllocatorSetUsed:299 : internal error: Failed to reserve port 49153 2020-10-12 02:11:33.852+0000: 6825: debug : qemuMigrationParamsReset:1206 : Resetting migration parameters 0x7fe784257c30, flags 0x59 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjBeginJobInternal:835 : Starting job: job=async nested agentJob=none asyncJob=none (vm=0x55f086c81ea0 name=cartridge, current job=none agentJob=none async=migration in) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjBeginJobInternal:887 : Started job: async nested (async=migration in vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjEnterMonitorInternal:5590 : Entering monitor (mon=0x7fe784255020 vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuMonitorSetMigrationCapabilities:3853 : mon:0x7fe784255020 vm:0x55f086c81ea0 fd:30 2020-10-12 02:11:33.852+0000: 6825: info : qemuMonitorSend:944 : QEMU_MONITOR_SEND_MSG: mon=0x7fe784255020 msg={"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability> fd=-1 ...
Full logs: destination: https://drive.google.com/file/d/1g986SbSVijvwZd8d7xDrJwo_AmKH-JnV/view?usp=s...
source: https://drive.google.com/file/d/1lsV2EOBxF7xH5-lgz2Psh9YSkOePXvAd/view?usp=s...
On target host, nothing is listening on the target port:
ratchet /var/log/libvirt/qemu # netstat -lnp | grep 49153 | wc -l 0
and nc -l 49153 succeeds without issues.
Do you have any suggestions on how to proceed here?
On 10/12/20 4:46 AM, Vjaceslavs Klimovs wrote:
On libvirt 6.8.0 and qemu 5.1.0, when trying to live migrate "error: internal error: Failed to reserve port" error is received and migration does not succeed:
virsh # migrate cartridge qemu+tls://ratchet.lan/system --live --persistent --undefinesource --copy-storage-all --verbose error: internal error: Failed to reserve port 49153
virsh #
Sorry for not replying earlier. But this is a clear libvirt bug and I think it's a regression introduced by the following commit: https://gitlab.com/libvirt/libvirt/-/commit/e74d627bb3b The problem is, if you have two or more disks that need to be copied over to the destination, the @server_started variable is not set after the first iteration of the "for (i = 0; i < vm->def->ndisks; i++)" loop. I think this should be the fix: diff --git i/src/qemu/qemu_migration.c w/src/qemu/qemu_migration.c index 2f5d61f8e7..6f764b0c73 100644 --- i/src/qemu/qemu_migration.c +++ w/src/qemu/qemu_migration.c @@ -479,9 +479,11 @@ qemuMigrationDstStartNBDServer(virQEMUDriverPtr driver, QEMU_ASYNC_JOB_MIGRATION_IN) < 0) goto cleanup; - if (!server_started && - qemuMonitorNBDServerStart(priv->mon, &server, tls_alias) < 0) - goto exit_monitor; + if (!server_started) { + if (qemuMonitorNBDServerStart(priv->mon, &server, tls_alias) < 0) + goto exit_monitor; + server_started = true; + } if (qemuBlockExportAddNBD(vm, diskAlias, disk->src, diskAlias, true, NULL) < 0) goto exit_monitor; Can you please give it a try? If it fixes the issue, I'll post it as a patch. Thanks, Michal
On 10/26/20 9:39 AM, Michal Privoznik wrote:
On 10/12/20 4:46 AM, Vjaceslavs Klimovs wrote:
On libvirt 6.8.0 and qemu 5.1.0, when trying to live migrate "error: internal error: Failed to reserve port" error is received and migration does not succeed:
virsh # migrate cartridge qemu+tls://ratchet.lan/system --live --persistent --undefinesource --copy-storage-all --verbose error: internal error: Failed to reserve port 49153
virsh #
Sorry for not replying earlier. But this is a clear libvirt bug and I think it's a regression introduced by the following commit:
https://gitlab.com/libvirt/libvirt/-/commit/e74d627bb3b
The problem is, if you have two or more disks that need to be copied over to the destination, the @server_started variable is not set after the first iteration of the "for (i = 0; i < vm->def->ndisks; i++)" loop. I think this should be the fix:
Actually, you will need a second patch too. Here's the series: https://www.redhat.com/archives/libvir-list/2020-October/msg01358.html Michal
Thank you for looking into this. Confirmed that with the patch the migration succeeds. On Mon, Oct 26, 2020 at 3:43 AM Michal Privoznik <mprivozn@redhat.com> wrote:
On 10/26/20 9:39 AM, Michal Privoznik wrote:
On 10/12/20 4:46 AM, Vjaceslavs Klimovs wrote:
On libvirt 6.8.0 and qemu 5.1.0, when trying to live migrate "error: internal error: Failed to reserve port" error is received and migration does not succeed:
virsh # migrate cartridge qemu+tls://ratchet.lan/system --live --persistent --undefinesource --copy-storage-all --verbose error: internal error: Failed to reserve port 49153
virsh #
Sorry for not replying earlier. But this is a clear libvirt bug and I think it's a regression introduced by the following commit:
https://gitlab.com/libvirt/libvirt/-/commit/e74d627bb3b
The problem is, if you have two or more disks that need to be copied over to the destination, the @server_started variable is not set after the first iteration of the "for (i = 0; i < vm->def->ndisks; i++)" loop. I think this should be the fix:
Actually, you will need a second patch too. Here's the series:
https://www.redhat.com/archives/libvir-list/2020-October/msg01358.html
Michal
participants (2)
-
Michal Privoznik -
Vjaceslavs Klimovs