Just a brief update:
As soon as I umount cephfs virsh is able to talk to libvirtd.
I tested the cephfs with:
df (no problem)
dd if=/dev/zero of=/cephstorage/a.img bs=1G count=1 oflag=dsync (no
problem created random1G file, no I/O issues.)
After mounting cephfs and restarting libvirtd "virsh" hangs again.
Obviously virsh and libvirtd don't like the cephfs mount.
I am just starting to debug the potential problem with cephfs and
libvirtd/virsh.
I originally noted when this problem occurred on two hypervisors the
problem occurred a couple days a part which matched some updates that took
place. I have not tried rolling back patches yet.
I am curious if anyone uses cephfs filesystem and had similar problems
recently.
I will update the form if I find a solution.
Thanks
jerry
Jerry Buburuz
Just found my issue.
After I removed the cephfs mounts it worked!
I will debug ceph.
I assumed because I could touch files on mounted cephfs it was working.
Now virsh list works!
thanks
jerry
Lars Kellogg-Stedman
> On Tue, Jun 06, 2023 at 04:56:38PM -0400, Jerry Buburuz wrote:
>> Recently both virsh stopped talking to the libvirtd. Both stopped
>> within
>> a
>> few days of each other.
>
> I've run into exactly the same problem.
>
> I'm running libvirt (libvirt-9.0.0-3.fc38.x86_64) on Fedora 38. On
> Fedora, libvirtd is configured by default to use socket activation and
> is run with the `--timeout 120` option.
>
> After some recent upgrades, I'm seeing the exact same symptoms that
> Jerry described -- virsh commands simply get stuck at same call to
> `poll()`.
>
> It looks like libvirtd is either crashing or failing to start, because
> when virsh is in this state the `libvirtd` process isn't running. This
> makes it *sound* like a systemd problem, but I'm not seeing errors
> anywhere -- either from libvirtd or from systemd.
>
> I've worked around the problem locally by re-configuring libvirtd to
> run persistently rather than using socket activation:
>
> systemctl disable --now libvirtd{,-ro,-admin}.socket
>
> cat > /etc/systemd/system/libvirtd.service.d/override.conf <<EOF
> [Service]
> EnvironmentFile=
> EOF
>
> systemctl restart libvirtd
>
> Package versions in case this helps correlate something:
>
> - libvirt-9.0.0-3.fc38.x86_64
> - systemd-253.5-1.fc38.x86_64
> - kernel-6.3.6-200.fc38.x86_64
>
> Libvirt uri: qemu:///system
>
> --
> Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {irc,twitter,github}
>
http://blog.oddbit.com/ | N1LKS
>
>