Libvirt and Ceph: libvirtd tries to open random RBD images
Hello Users, We're using libvirt with KVM and the orchestrator is Cloudstack. I raised the issue already at Cloudstack at https://github.com/apache/cloudstack/issues/8211 but appears to be at libvirtd. Below is our environment & issue description: It appears that one of our Cloudstack KVM clusters having 8 hosts is having the issue. We have HCI on these 8 hosts and there are approximately 700+ VMs running. But strange enough, there are these logs like below on hosts. Oct 25 13:38:11 hv-01 libvirtd[9464]: failed to open the RBD image '087bb114-448a-41d2-9f5d-6865b62eed15': No such file or directory Oct 25 20:35:22 hv-01 libvirtd[9464]: failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory Oct 26 09:48:33 hv-01 libvirtd[9464]: failed to open the RBD image 'a3fe82f8-afc9-4604-b55e-91b676514a18': No such file or directory Oct 26 10:38:17 hv-01 libvirtd[9464]: End of file while reading data: Input/output error We've got DNS servers on which there is an`A` record resolving to all the IPv4 Addresses of 8 monitors and there have not been any issues with the DNS resolution. But the issue of "failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory" gets more weird because the VM that is making use of that RBD image lets say "087bb114-448a-41d2-9f5d-6865b62eed15" is running on altogether different host like "hv-06". On further inspection of that specific Virtual Machine, it has been running on that host "hv-06" for more than 4 months or so (looked at "Last updated" field). Fortunately, the Virtual Machine has no issues and has been running since then. Thanks, Jayanth
On Mon, Nov 20, 2023 at 12:41:58PM +0530, Jayanth Reddy wrote:
Hello Users, We're using libvirt with KVM and the orchestrator is Cloudstack. I raised the issue already at Cloudstack at https://github.com/apache/cloudstack/issues/8211 but appears to be at libvirtd. Below is our environment & issue description:
It appears that one of our Cloudstack KVM clusters having 8 hosts is having the issue. We have HCI on these 8 hosts and there are approximately 700+ VMs running. But strange enough, there are these logs like below on hosts.
Oct 25 13:38:11 hv-01 libvirtd[9464]: failed to open the RBD image '087bb114-448a-41d2-9f5d-6865b62eed15': No such file or directory Oct 25 20:35:22 hv-01 libvirtd[9464]: failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory Oct 26 09:48:33 hv-01 libvirtd[9464]: failed to open the RBD image 'a3fe82f8-afc9-4604-b55e-91b676514a18': No such file or directory Oct 26 10:38:17 hv-01 libvirtd[9464]: End of file while reading data: Input/output error
We've got DNS servers on which there is an`A` record resolving to all the IPv4 Addresses of 8 monitors and there have not been any issues with the DNS resolution. But the issue of "failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory" gets more weird because the VM that is making use of that RBD image lets say "087bb114-448a-41d2-9f5d-6865b62eed15" is running on altogether different host like "hv-06". On further inspection of that specific Virtual Machine, it has been running on that host "hv-06" for more than 4 months or so (looked at "Last updated" field). Fortunately, the Virtual Machine has no issues and has been running since then.
This is an indication that you have a storage pool defined with the RBD backend. Libvirt tries to enumerate all the images in the pool, and has to open the RBD image to query its size. I won't be trying to read/write the payload of the image. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hello Daniel, Thank you for the response. Please also let me know why libvirtd says "No such file or directory" even though the block images exist and are healthy. Thanks, Jayanth ________________________________ From: Daniel P. Berrangé <berrange@redhat.com> Sent: Monday, November 20, 2023 2:36:21 PM To: Jayanth Reddy <jayanthreddy5666@gmail.com> Cc: users@lists.libvirt.org <users@lists.libvirt.org> Subject: Re: Libvirt and Ceph: libvirtd tries to open random RBD images On Mon, Nov 20, 2023 at 12:41:58PM +0530, Jayanth Reddy wrote:
Hello Users, We're using libvirt with KVM and the orchestrator is Cloudstack. I raised the issue already at Cloudstack at https://github.com/apache/cloudstack/issues/8211 but appears to be at libvirtd. Below is our environment & issue description:
It appears that one of our Cloudstack KVM clusters having 8 hosts is having the issue. We have HCI on these 8 hosts and there are approximately 700+ VMs running. But strange enough, there are these logs like below on hosts.
Oct 25 13:38:11 hv-01 libvirtd[9464]: failed to open the RBD image '087bb114-448a-41d2-9f5d-6865b62eed15': No such file or directory Oct 25 20:35:22 hv-01 libvirtd[9464]: failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory Oct 26 09:48:33 hv-01 libvirtd[9464]: failed to open the RBD image 'a3fe82f8-afc9-4604-b55e-91b676514a18': No such file or directory Oct 26 10:38:17 hv-01 libvirtd[9464]: End of file while reading data: Input/output error
We've got DNS servers on which there is an`A` record resolving to all the IPv4 Addresses of 8 monitors and there have not been any issues with the DNS resolution. But the issue of "failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory" gets more weird because the VM that is making use of that RBD image lets say "087bb114-448a-41d2-9f5d-6865b62eed15" is running on altogether different host like "hv-06". On further inspection of that specific Virtual Machine, it has been running on that host "hv-06" for more than 4 months or so (looked at "Last updated" field). Fortunately, the Virtual Machine has no issues and has been running since then.
This is an indication that you have a storage pool defined with the RBD backend. Libvirt tries to enumerate all the images in the pool, and has to open the RBD image to query its size. I won't be trying to read/write the payload of the image. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (2)
-
Daniel P. Berrangé -
Jayanth Reddy