Hello Users,

We're using libvirt with KVM and the orchestrator is Cloudstack. I raised the issue already at Cloudstack at https://github.com/apache/cloudstack/issues/8211 but appears to be at libvirtd. Below is our environment & issue description:

It appears that one of our Cloudstack KVM clusters having 8 hosts is having the issue. We have HCI on these 8 hosts and there are approximately 700+ VMs running. But strange enough, there are these logs like below on hosts.


Oct 25 13:38:11 hv-01 libvirtd[9464]: failed to open the RBD image '087bb114-448a-41d2-9f5d-6865b62eed15': No such file or directory
Oct 25 20:35:22 hv-01 libvirtd[9464]: failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory
Oct 26 09:48:33 hv-01 libvirtd[9464]: failed to open the RBD image 'a3fe82f8-afc9-4604-b55e-91b676514a18': No such file or directory
Oct 26 10:38:17 hv-01 libvirtd[9464]: End of file while reading data: Input/output error


We've got DNS servers on which there is an`A` record resolving to all the IPv4 Addresses of 8 monitors and there have not been any issues with the DNS resolution. But the issue of "failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory" gets more weird because the VM that is making use of that RBD image lets say "087bb114-448a-41d2-9f5d-6865b62eed15" is running on altogether different host like "hv-06". On further inspection of that specific Virtual Machine, it has been running on that host "hv-06" for more than 4 months or so (looked at "Last updated" field). Fortunately, the Virtual Machine has no issues and has been running since then.

Thanks,

Jayanth