6:42 a.m.
Hi everybody,
my background: I'm doing Xen since 10++ years, many years with DRBD for
high availability, since some time I'm using preferable GlusterFS with
FUSE as replicated storage, where I place the image-files for the vms.
In my current project we started (successfully) with Xen/GlusterFS too,
but because the provider, where we placed the servers, uses widely CEPH,
we decided to switch too.
Unfortunately I'm new to CEPH, but with help of a technician, we have
running a 3 node CEPH-cluster now, that seems to work fine.
Hardware:
- Xeons, 24 Cores, 256 GB RAM,
2x 240 GB system-SSDs RAID1, 4x 1,92 TB data-SSDs (no RAID)
Software we are using:
- CentOS 7.5.1804
- Kernel: 4.9.86-30.el7 @centos-virt-xen-48
- Xen: 4.8.3-5.el7 @centos-virt-xen-48
- libvirt-xen: 4.1.0-2.xen48.el7 @centos-virt-xen-48
- Ceph: 2:12.2.5-0.el7 @Ceph
CEPH itself works, Xen too and even libvirt (I'm quite new in this too,
normally I use the XL-tools to manage my VMs) ;-) I can manually map an
RBD and start a PV-VM via libvirt, after modifying the XML-config and
defined the VM with libvirt.
That is what happens:
# virsh define xml/testvm3.xml
Domain testvm3 defined from xml/testvm3.xml
# virsh start --console testvm3
error: Failed to start domain testvm3
error: internal error: libxenlight failed to create new domain 'testvm3'
xml/testvm3.xml:
-> ...
<devices>
<disk type='network' device='disk'>
<source protocol='rbd' name='devel-pool/testvm3.rbd'>
<host name="10.20.30.1" port="6789"/>
<host name="10.20.30.2" port="6789"/>
<host name="10.20.30.3" port="6789"/>
</source>
<auth username='libvirt'>
<secret type='ceph' uuid='07f3a0fe-9178-xxxxxxxxx'/>
</auth>
<target dev='xvda' bus='xen'/>
</disk>
...
Logs:
/var/log/libvirt/libxl/libxl-driver.log:
-> ...
2018-05-27 11:24:36.666+0000: libxl:
libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult
logfile /var/log/xen/bootloader.14.log
2018-05-27 11:24:36.666+0000: libxl:
libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [183216]
exited with error status 1
2018-05-27 11:24:36.667+0000: libxl:
libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3
/var/log/xen/bootloader.14.log:
->
Traceback (most recent call last):
File "/usr/lib64/xen/bin/pygrub", line 896, in <module>
part_offs = get_partition_offsets(file)
File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets
image_type = identify_disk_image(file)
File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image
fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory:
'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789'
So the Xen-Python-script, that searches for the kernel/bootloader of the
VM, does not find the imagefile. But if I grab the above output and use
it with qemu, it does find the blockdevice:
# qemu-img info
"rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789"
->
image:
rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\;none:mon_host=10.20.30.1\:6789\;10.20.30.2\:6789\;10.20.30.3\:6789
file format: raw
virtual size: 16G (17179869184 bytes)
disk size: unavailable
Thus it seems, that CEPH-auth works and the VM-XML-config is interpreted
correct by libvirt. But the next step fails, when libvirt should pass
the blockdevice to Xen?
Anybody out there who uses a similar configuration? Any hints how to
debug this? Unfortunately I do not have anymore logs to look at.
Thank you very much,
--
Kind regards,
thg