Re: Error Cannot acquire state change lock from remoteDispatchDomainMigratePrepare3Params during live migration of domains
On 3/7/24 10:51, Martin Kletzander wrote:
On Wed, Mar 06, 2024 at 05:17:36PM +0100, Christian Rohmann via
Users
wrote:
> Hallo libvirt-users!
>
Hi, I'll try to reply in the simplest possible way.
> we observe lock-ups / timeouts with in prometheus-libvirt-exporter
> (https://github.com/inovex/prometheus-libvirt-exporter) when
> libvirt is live-migrating domains:
>
>> Timed out during operation: cannot acquire state change lock (held by
>> monitor=remoteDispatchDomainMigratePrepare3Params)
>
>
> All of the source code can be found at:
>
https://github.com/inovex/prometheus-libvirt-exporter/blob/master/pkg/exp....
> Basically the error happens when DomainMemoryStats or other operational
> domain info is queried via the libvirt socket.
>
Yes, the domain is being modified by the migration, so it is locked.
While this is true, the "lock" - or job I should rather say is an async
one, meaning a QUERY job can be acquired. It's only MODIFY job that
should wait in the queue.
What's rather weird is - the thread holding the job is 'MigratePrepare'
which usually isn't that long.
@Christian, what is the libvirt version? Are you able to reproduce with
either libvirt-10.1.0 or (even better) current master?
Michal