Hi all, At the moment, RBD storage pools in libvirt must be supplied with a list of Ceph monitor addresses, using <host> elements in the pool's source definition. Ceph itself has a configuration file, and this is used by default by all Ceph command-line utilities. This file can contain the monitor addresses for the cluster, as well as a bunch of other useful options (e.g. for tuning and debugging). I think it would be nice if libvirt were able to load in this file when starting RBD storage pools. Before I send some patches through. however, I thought I'd better check to see whether my approach is sound. First, I am not keen on having libvirt get librados to load the configuration file automatically. librados actually uses a search path to find the configuration file, and that path includes silly things like the current working directory. Since it can be told to load a single file, I think it would be better if it were made explicit in the storage pool XML, i.e.: <pool type="rbd"> <name>rbd</name> <source> <name>rbd</name> <config file="/etc/ceph/ceph.conf"/> <auth username="user" type="ceph"> <secret usage="mycephcluster"/> </auth> </source> </pool> <config> would be able to be used in addition to, or as an alternative to, a list of <host> elements. Would something along these lines this be suitable? Would it be better to use the <config> element's text content as the filename, rather than use an attribute? I'm not sure what style guidelines there are for something like this. The second part is of course to make a similar change to RBD-based domain disk definitions, i.e.: ... <disk type="network"> <driver name="qemu" type="raw"/> <source protocol="rbd" name="pool/volume"> <config file="/etc/ceph/ceph.conf"/> </source> <target dev="vda" bus="virtio"/> <auth username="user"> <secret type="ceph" usage="mycephcluster"/> </auth> </disk> ... Again, <config> could be used instead of or alongside some <host> elements. This is where it gets a little tricky. At the moment, <host> in a disk's source definition is entirely optional. Furthermore, QEMU _always_ loads a Ceph configuration file -- either one supplied as a "conf" argument for the block device, or one found through the search path mentioned earlier. The only way to suppress this is to pass conf=/dev/null... but for backwards-compatibility (users may be relying on QEMU's use of the search path), I don't think we can do this now. There's one final gotcha in all of this: if QEMU is given both a "conf" argument and a "mon_addr" argument, only the latter will take effect. This means if both <config> and <host> are supplied, then the <host> elements will override any monitor addresses from the configuration file. For consistency, I intend to make an RBD storage pool have the same behaviour. However, would it perhaps be better if the user could only choose _either_ <config> or a list of <host> elements? Personally, I don't think it's a big deal if the behaviour is clearly documented -- being able to load options from a config file while still defining hosts in the libvirt XML could be useful. Anyway, before I send my patches through I'm interested in hearing people's thoughts on this. All sound sane? Too intrusive? A waste of time? :-)

On Fri, 1 Nov 2013, Daniel P. Berrange wrote: > We have always taken the position that we do not want to rely on host > configuration in this way. The goal of the XML configs is that they > fully describe the functional setup of the resource in question. This > is to ensure that if you put the same XML config on two different hosts > you can be sure that they will operate in the same way. If you leave out > a bunch of config information and rely on the host ceph.conf file, then > you can no longer ever be sure if two hosts are configured the same way > with libvirt. I suspected that might be the case -- half the reason I sent my email, really! If it's desireable to not rely on any host configuration at all, should we be explicitly be passing conf=/dev/null to QEMU when setting up a RBD device?

As I mentioned before, without that QEMU will implicitly try to find a system ceph.conf file using a built-in librados search path. Would this actually be backwards-incompatible change given it was never documented by libvirt?

On 11/01/2013 08:31 AM, Michael Chapman wrote: > As I mentioned before, without that QEMU will implicitly try to > find a system ceph.conf file using a built-in librados search path. > Would this actually be backwards-incompatible change given it was never > documented by libvirt? The old behavior is broken, so we can bill this as a bug fix (previously, qemu would behave differently than what the XML defined, which is not supposed to happen) rather than a backwards-incompatible change. Can you propose a patch in time for inclusion in 1.1.4?