9:13 a.m.
Dear libvirt community,
Using recent Ubuntu Stein Cloud Packages, we are observing random
libvirtd live-migration crashes on the target host.
Libvirt is having a SEGFAULT with the qemu driver. Transferring block
devices usually works without issues.
However, the following memory transfer is causing the target libvirtd
randomly to close down its socket, resulting in a roll-backed migration
process.I can reproduce this with large VMs, which have a large memory pool.
The last error message we see in libvirt logs is:
error : qemuMonitorJSONCommandWithFd:315 : internal error: Missing
monitor reply object
With this, libvirt segfaults and restarts.
Before we encountered this issue, we used an older nova-compute package
(19.0.3).
Not sure if this made a difference with usage of libvirtd-api.
After upgrade, we also see a lot of recurring errors during migration:
warning : qemuDomainObjBeginJobInternal:7044 : Cannot start job (query,
none, none) for domain instance-00008f56; current job is (none, none,
migration in) owned by (0 <null>, 0 <null>, 0
remoteDispatchDomainMigratePrepare3Params (flags=0x809b)) for (0s, 0s,
14834s)
error : qemuDomainObjBeginJobInternal:7066 : Timed out during operation:
cannot acquire state change lock (held by
monitor=remoteDispatchDomainMigratePrepare3Params)
They don't abort the running migration process, but spam every minute to
the systemd journal.
Source and destination run the same packages:
Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-99-generic x86_64)
OpenStack Stein (Ubuntu Cloud Archive)
Libvirt+QEMU_x86
keystone-common 2:15.0.1-0ubuntu1~cloud0
libvirt-daemon 5.0.0-1ubuntu2.6~cloud0
qemu-system-x86 1:3.1+dfsg-2ubuntu3.7~cloud0
neutron-linuxbridge-agent 2:14.2.0-0ubuntu1~cloud0
neutron-plugin-ml2 2:14.2.0-0ubuntu1~cloud0
nova-compute 2:19.2.0-0ubuntu1~cloud0
nova-compute-libvirt 2:19.2.0-0ubuntu1~cloud0
I have attached source/destination debug logs from libvirtd and
nova-compute here:
https://denzelx.ddns.net/index.php/s/KPJ7vv4aTcb69XD
Any help would be nice!
Best Regards
--
M.Sc Alex Walender
de.NBI Cloud Bielefeld Administrator
Center for Biotechnology (CeBiTec)
University of Bielefeld
33594 Bielefeld
Germany
room: M3-118
phone: +49 (521) 106 2907
10:03 a.m.
On Thu, Jul 30, 2020 at 16:13:09 +0200, Alex Walender wrote:
Dear libvirt community,
Using recent Ubuntu Stein Cloud Packages, we are observing random
libvirtd live-migration crashes on the target host.
Libvirt is having a SEGFAULT with the qemu driver. Transferring block
devices usually works without issues.
However, the following memory transfer is causing the target libvirtd
randomly to close down its socket, resulting in a roll-backed migration
process.I can reproduce this with large VMs, which have a large memory pool.
Could you please also attach the backtrace of the libvirtd process once
it crashes? That would be helpful.
6:38 a.m.
Hi Peter,
I have attached a trace via coredump.
Best Regards
Am 30.07.20 um 17:03 schrieb Peter Krempa:
On Thu, Jul 30, 2020 at 16:13:09 +0200, Alex Walender wrote:
> Dear libvirt community,
>
>
> Using recent Ubuntu Stein Cloud Packages, we are observing random
> libvirtd live-migration crashes on the target host.
> Libvirt is having a SEGFAULT with the qemu driver. Transferring block
> devices usually works without issues.
> However, the following memory transfer is causing the target libvirtd
> randomly to close down its socket, resulting in a roll-backed migration
> process.I can reproduce this with large VMs, which have a large memory pool.
Could you please also attach the backtrace of the libvirtd process once
it crashes? That would be helpful.
--
M.Sc Alex Walender
de.NBI Cloud Bielefeld Administrator
Center for Biotechnology (CeBiTec)
University of Bielefeld
33594 Bielefeld
Germany
room: M3-118
phone: +49 (521) 106 2907
1456
days inactive
1457
days old
3 comments
3 participants
participants (3)
-
Alex Walender -
Michal Privoznik -
Peter Krempa