Just a brief update: As soon as I umount cephfs virsh is able to talk to libvirtd. I tested the cephfs with: df (no problem) dd if=/dev/zero of=/cephstorage/a.img bs=1G count=1 oflag=dsync (no problem created random1G file, no I/O issues.) After mounting cephfs and restarting libvirtd "virsh" hangs again. Obviously virsh and libvirtd don't like the cephfs mount. I am just starting to debug the potential problem with cephfs and libvirtd/virsh. I originally noted when this problem occurred on two hypervisors the problem occurred a couple days a part which matched some updates that took place. I have not tried rolling back patches yet. I am curious if anyone uses cephfs filesystem and had similar problems recently. I will update the form if I find a solution. Thanks jerry Jerry Buburuz
Just found my issue.
After I removed the cephfs mounts it worked!
I will debug ceph.
I assumed because I could touch files on mounted cephfs it was working.
Now virsh list works!
thanks jerry
Lars Kellogg-Stedman
On Tue, Jun 06, 2023 at 04:56:38PM -0400, Jerry Buburuz wrote:
Recently both virsh stopped talking to the libvirtd. Both stopped within a few days of each other.
I've run into exactly the same problem.
I'm running libvirt (libvirt-9.0.0-3.fc38.x86_64) on Fedora 38. On Fedora, libvirtd is configured by default to use socket activation and is run with the `--timeout 120` option.
After some recent upgrades, I'm seeing the exact same symptoms that Jerry described -- virsh commands simply get stuck at same call to `poll()`.
It looks like libvirtd is either crashing or failing to start, because when virsh is in this state the `libvirtd` process isn't running. This makes it *sound* like a systemd problem, but I'm not seeing errors anywhere -- either from libvirtd or from systemd.
I've worked around the problem locally by re-configuring libvirtd to run persistently rather than using socket activation:
systemctl disable --now libvirtd{,-ro,-admin}.socket
cat > /etc/systemd/system/libvirtd.service.d/override.conf <<EOF [Service] EnvironmentFile= EOF
systemctl restart libvirtd
Package versions in case this helps correlate something:
- libvirt-9.0.0-3.fc38.x86_64 - systemd-253.5-1.fc38.x86_64 - kernel-6.3.6-200.fc38.x86_64
Libvirt uri: qemu:///system
-- Lars Kellogg-Stedman <lars@redhat.com> | larsks @ {irc,twitter,github} http://blog.oddbit.com/ | N1LKS