3:31 p.m.
Hello,
I setup a Ceph Cluster on my system in hopes of using it with libvirtd,
however I'm finding myself unable to have libvirtd make the connection.
[root@server ~]# virsh pool-start "${CEPH_POOL}"
error: Failed to start pool libvirt-pool
error: failed to connect to the RADOS monitor on: localhost:6789,: No
such file or directory
I know that rbd is working as I am able to use it with the rbd utility.
[root@server ~]# rbd list libvirt-pool
Kolab
[root@server ~]# rbd device map libvirt-pool/Kolab
/dev/rbd0
[root@server ~]# lsblk /dev/rbd0
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
rbd0 253:0 0 20G 0 disk
├─rbd0p1 253:1 0 1G 0 part
└─rbd0p2 253:2 0 19G 0 part
The libvirtd logs are not providing any useful information on the issue,
I tried changing log-level to 1 and enabling the log_filters/adding
"1:rbd" to the log filters.
I have tried both the libvirt version 7.3.0 in my repository and the
bleeding edge with a compile from git. Both returns the same error.
Any help in troubleshooting this issue? I'm somewhat new when it comes
to all of this.
Thank you,
James Coleman
8:48 p.m.
On Sun, May 23, 2021 at 4:33 AM Mr. Gecko <grmrgecko(a)gmail.com> wrote:
Hello,
I setup a Ceph Cluster on my system in hopes of using it with libvirtd,
however I'm finding myself unable to have libvirtd make the connection.
[root@server ~]# virsh pool-start "${CEPH_POOL}"
error: Failed to start pool libvirt-pool
error: failed to connect to the RADOS monitor on: localhost:6789,: No
such file or directory
I know that rbd is working as I am able to use it with the rbd utility.
[root@server ~]# rbd list libvirt-pool
Kolab
[root@server ~]# rbd device map libvirt-pool/Kolab
/dev/rbd0
[root@server ~]# lsblk /dev/rbd0
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
rbd0 253:0 0 20G 0 disk
├─rbd0p1 253:1 0 1G 0 part
└─rbd0p2 253:2 0 19G 0 part
The libvirtd logs are not providing any useful information on the issue,
I tried changing log-level to 1 and enabling the log_filters/adding
"1:rbd" to the log filters.
I have tried both the libvirt version 7.3.0 in my repository and the
bleeding edge with a compile from git. Both returns the same error.
You can check the following:
1. Make sure the ceph mon node are running on the localhost:6789
2. Check if the ceph authorization is enabled on the ceph cluster. Check if
authorization is
used in rbd pool xml.
You can provide more details on the rbd pool xml, ceph conf, and the
version of librbd1(client side), ceph(server side).
Any help in troubleshooting this issue? I'm somewhat new when it
comes
to all of this.
Thank you,
James Coleman
9:23 a.m.
On 5/23/21 8:48 PM, Han Han wrote:
You can check the following:
1. Make sure the ceph mon node are running on the localhost:6789
root@server ~# netstat -pan | grep 6789
tcp 0 0 10.0.0.5:6789 0.0.0.0:* LISTEN
2087/ceph-mon
2. Check if the ceph authorization is enabled on the ceph cluster.
Check if authorization is
used in rbd pool xml.
You can provide more details on the rbd pool xml, ceph conf, and the
version of librbd1(client side), ceph(server side).
root@server ~# ceph auth get client.libvirt
exported keyring for client.libvirt
[client.libvirt]
key = AQDtM6lgXXXXXXXXpebpIoIKaFXQSTAg==
caps mon = "profile rbd"
caps osd = "profile rbd pool=libvirt-pool"
root@server ~# virsh secret-list
UUID Usage
--------------------------------------------------------------------
f290cb3a-a9de-48d7-89ce-09311eadff4a ceph client.libvirt secret
root@server ~# virsh secret-dumpxml f290cb3a-a9de-48d7-89ce-09311eadff4a
<secret ephemeral='no' private='no'>
<uuid>f290cb3a-a9de-48d7-89ce-09311eadff4a</uuid>
<usage type='ceph'>
<name>client.libvirt secret</name>
</usage>
</secret>
root@server ~# virsh pool-dumpxml libvirt-pool
<pool type='rbd'>
<name>libvirt-pool</name>
<uuid>5ae8abb9-31ab-44c0-a708-072b95182018</uuid>
<capacity unit='bytes'>0</capacity>
<allocation unit='bytes'>0</allocation>
<available unit='bytes'>0</available>
<source>
<host name='localhost' port='6789'/>
<name>libvirt-pool</name>
<auth type='ceph' username='libvirt'>
<secret uuid='f290cb3a-a9de-48d7-89ce-09311eadff4a'/>
</auth>
</source>
</pool>
root@server ~# ceph versions
{
"mon": {
"ceph version 15.2.12
(ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 2
},
"mgr": {
"ceph version 15.2.12
(ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 2
},
"osd": {
"ceph version 15.2.12
(ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 2
},
"mds": {},
"rgw": {
"ceph version 15.2.12
(ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 1
},
"overall": {
"ceph version 15.2.12
(ce065eabfa5ce81323b009786bdf5bb03127cbe1) octopus (stable)": 7
}
}
root@server ~# pkgfile /usr/lib/librbd.so.1.12.0
community/ceph-libs
root@server ~# pacsearch ceph-libs
community/ceph-libs 15.2.12-1 [installed]
Distributed, fault-tolerant storage platform delivering object,
block, and file system
Sorry I should have included this information in my original email. I
followed this guide
(https://blog.modest-destiny.com/posts/kvm-libvirt-add-ceph-rbd-pool/)
to setup the above configuration and I also tried following the guide at
https://docs.ceph.com/en/latest/rbd/libvirt/
12:41 p.m.
Ok, so I figured out my problem after digging deeper into
troubleshooting. My issue was just that ceph monitor is not listening to
port 6789 on all interfaces. Its just listening to it on the network
address 10.0.0.5. Changing the pool configuration to account for that
fixed my issue. I wrote a simple c program based off documentation at
https://docs.ceph.com/en/latest/rados/api/librados/ which is where it
hit me after trying to use `rbd -m localhost:6789 list libvirt-pool`
1314
days inactive
1316
days old
3 comments
2 participants
participants (2)
-
Han Han -
Mr. Gecko