Error Cannot acquire state change lock from
remoteDispatchDomainMigratePrepare3Params during live migration of domains
by Christian Rohmann
Hallo libvirt-users!
we observe lock-ups / timeouts with in prometheus-libvirt-exporter
(https://github.com/inovex/prometheus-libvirt-exporter) when
libvirt is live-migrating domains:
> Timed out during operation: cannot acquire state change lock (held by
> monitor=remoteDispatchDomainMigratePrepare3Params)
All of the source code can be found at:
https://github.com/inovex/prometheus-libvirt-exporter/blob/master/pkg/exp....
Basically the error happens when DomainMemoryStats or other operational
domain info is queried via the libvirt socket.
1) We are actually using the read-only socket at
'/var/run/libvirt/libvirt-sock-ro', so there should not be any locking
required.
Is there any way to not run into lock contention, like running a request
with some "nolock" indication?
2) This being reported as timeout waiting for the lock, what is the
timeout and would waiting a bit longer help?
Or is the lock active during the whole time a domain live migration is
running?
3) Is this in any way related to the type of migration? Tunneled vs.
native (https://libvirt.org/migration.html)?
4) Is there any indication that we could use to skip those domains (or
certain queries)?
The same issue was actually previously reported for another
implementation of a Prometheus exporter
(https://github.com/kumina/libvirt_exporter/issues/33).
Currently the exporter locks up or throws the mentioned timeout errors
during the the migration of 200 domains, 5 at a time.
It would be awesome to find a way to make this work as smooth as
possible, even during live migrations!
I am thankful for any insights into how the libvirt socket, the various
calls, the locking mechanisms or live migration modes work!
Regards
Christian
6 months, 1 week
How to configure the default connection for virsh
by Nir Soffer
We are using minikube vms, which require adding the user to to libvirt
group[1].
We use `virsh -c qemu:///system` for debugging these vms or related libvirt
networks.
Using virsh without specifying the '-c' argument is a common mistake that
leads to
trouble, for example creating the libvirt default network incorrectly.
I wonder if it is possible to configure virsh so it uses -c qemu:///system
by default?
We know that this is not great, but we don't know a better way to use
minikube
with the kvm2 driver. The minikube vms are started for creating a
development
environment and during CI, and they cannot require a password or any
complicated
setup.
[1] https://github.com/kubernetes/minikube/issues/3467
Nir
6 months, 2 weeks
Linking virtual sound cards in two virtual machines?
by Lars Kellogg-Stedman
I would like to create a virtual audio cable between two libvirt
guests such that sound output from guest A goes to sound input on
guest B, and vice versa. I figure the solution to this is using
something like the pulseaudio's "virtual sink", but I'm not entirely
sure what the syntax would look like (fwiw, I'm on Fedora 39, which
means libvirt 9.7.0, which means this has to be through libvirt's
pulseaudio driver rather than the pipewire driver, since the latter
isn't available in 9.7.0).
I started by creating a virtual sink (and yes, I realize that's only
half the process):
pactl load-module module-null-sink sink_name=vaudio0
And adding the following snippet to the domain xml for guest "node1.virt":
<audio id='1' type='pulseaudio'>
<input name='vaudio0' streamName='node1-in' latency='100'/>
<output name='vaudio0' streamName='node1-out' latency='100'/>
</audio>
But trying to start this guest results in:
error: Failed to start domain 'node1.virt'
error: internal error: process exited while connecting to monitor:
audio: Could not init `pa' audio driver
There doesn't appear to be any additional information in
/var/log/libvirt/qemu/node1.virt.log. I'm wondering if someone has an
example of what this might look like?
Thanks,
--
Lars Kellogg-Stedman <lars(a)redhat.com>
6 months, 2 weeks
'nconnect' nfs mount option support
by Abhisar Sinha
Hi,
Is there a way to specify the 'nconnect' nfs mount opt value in the XML
file used by 'virsh create-pool' command? If not, how hard would it be to
implement this functionality? Please advise.
Thanks,
Abhisar
6 months, 2 weeks
TLS Configuration with libvirt and Perfect Forward Secrecy
by Tommy.J.Mairo
Dear libvirt Users,
I'm seeking guidance on configuring TLS with libvirt to achieve perfect forward secrecy (PFS). While I followed the tutorial on setting up TLS with libvirt from the project wiki [1], I encountered an error when bringing up the legacy monolithic libvirt daemon.
The error message, "Certificate /etc/pki/libvirt/servercert.pem usage does not permit key encipherment," appeared even though I declared the "encryption_key" field in the certtool template file when generating the server certificate. Moreover, when I inspected the generated certificate with certtool, I discovered that it only had a "Digital signature" entry under X509v3 key usage. The deficiency in "Key Encipherment" usage could result from the key type specifications when generating the private key with the command "certtool --generate-privkey --key-type=ed25519 --curve=Ed25519 --sec-param=ultra > /etc/pki/libvirt/private/serverkey.pem". I prefer using elliptic-curve cryptography, specifically curve25519, over RSA, which the tutorial employs.
Although a digital signature-only certificate can still perform Diffie–Hellman exchanges with ECDSA, it cannot encrypt client-generated pre-master secrets. Nonetheless, a TLS session can still be established without pre-master secrets, avoiding session key leakage even if the decryption key is compromised in the future. This fulfills the PFS requirement, and I believe that a certificate without "Key Encipherment" usage contributes to the system's overall security.
Based on my understanding, I am seeking any insights or suggestions on whether there is a way to suppress the "certificate usage does not permit key encipherment" error and allow the legacy monolithic libvirt daemon to proceed.
Thanks,
Tommy.J.Mairo
[1] https://wiki.libvirt.org/TLSSetup.html
6 months, 3 weeks
