[libvirt-users] Setting RBD cache parameters for libvirt+qemu
I am running some qemu-kvm virtual machines via libvirt using Ceph RBD as the back-end storage. Today I was testing an update to libvirt-1.0.6 on one of my hosts and discovered that it includes this change: [libvirt] [PATCH] Forbid use of ':' in RBD pool names ...People are known to be abusing the lack of escaping in current libvirt to pass arbitrary args to QEMU. I am one of those people--I have been storing colon-separated arguments in my disk source 'name' attribute to control the RBD cache settings on a per-volume basis. Under libvirt-1.0.6 such VM's refuse to start: error: Failed to start domain foo error: unsupported configuration: ':' not allowed in RBD source volume name 'mypool/foo:rbd_cache_size=268435456:rbd_cache_max_dirty=134217728:rbd_cache_target_dirty=33554432:rbd_cache_max_dirty_age=5' So my question is how do I control the cache settings now? Thanks, JN
On 06/07/2013 02:41 PM, John Nielsen wrote:
I am running some qemu-kvm virtual machines via libvirt using Ceph RBD as the back-end storage. Today I was testing an update to libvirt-1.0.6 on one of my hosts and discovered that it includes this change: [libvirt] [PATCH] Forbid use of ':' in RBD pool names ...People are known to be abusing the lack of escaping in current libvirt to pass arbitrary args to QEMU.
I am one of those people--I have been storing colon-separated arguments in my disk source 'name' attribute to control the RBD cache settings on a per-volume basis. Under libvirt-1.0.6 such VM's refuse to start: error: Failed to start domain foo error: unsupported configuration: ':' not allowed in RBD source volume name 'mypool/foo:rbd_cache_size=268435456:rbd_cache_max_dirty=134217728:rbd_cache_target_dirty=33554432:rbd_cache_max_dirty_age=5'
So my question is how do I control the cache settings now?
You can set the cache mode with the cache attribute on the <driver> element inside a <disk> [1]. This works for rbd with QEMU 1.2 or later. QEMU will try to read /etc/ceph/ceph.conf by default as well, so you can put more specific settings in a [global] or [client.foo] section there. Note that if you enable writeback caching here, you need to include cache='writeback' in the xml as well, or qemu will not send flushes. Josh [1] http://libvirt.org/formatdomain.html#elementsDisks
On Jun 7, 2013, at 5:01 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
On 06/07/2013 02:41 PM, John Nielsen wrote:
I am running some qemu-kvm virtual machines via libvirt using Ceph RBD as the back-end storage. Today I was testing an update to libvirt-1.0.6 on one of my hosts and discovered that it includes this change: [libvirt] [PATCH] Forbid use of ':' in RBD pool names ...People are known to be abusing the lack of escaping in current libvirt to pass arbitrary args to QEMU.
I am one of those people--I have been storing colon-separated arguments in my disk source 'name' attribute to control the RBD cache settings on a per-volume basis. Under libvirt-1.0.6 such VM's refuse to start: error: Failed to start domain foo error: unsupported configuration: ':' not allowed in RBD source volume name 'mypool/foo:rbd_cache_size=268435456:rbd_cache_max_dirty=134217728:rbd_cache_target_dirty=33554432:rbd_cache_max_dirty_age=5'
So my question is how do I control the cache settings now?
You can set the cache mode with the cache attribute on the <driver> element inside a <disk> [1]. This works for rbd with QEMU 1.2 or later.
QEMU will try to read /etc/ceph/ceph.conf by default as well, so you can put more specific settings in a [global] or [client.foo] section there. Note that if you enable writeback caching here, you need to include cache='writeback' in the xml as well, or qemu will not send flushes.
Thanks for the quick reply. Is it safe to assume that the "foo" in "client.foo" should match the "auth username" attribute from the XML? If I want different cache settings for different VM's or volumes on the same host is there a way to specify that any more? Can I specify multiple profiles in ceph.conf or point different qemu instances to different ceph.conf files? JN
On 06/07/2013 04:18 PM, John Nielsen wrote:
On Jun 7, 2013, at 5:01 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
On 06/07/2013 02:41 PM, John Nielsen wrote:
I am running some qemu-kvm virtual machines via libvirt using Ceph RBD as the back-end storage. Today I was testing an update to libvirt-1.0.6 on one of my hosts and discovered that it includes this change: [libvirt] [PATCH] Forbid use of ':' in RBD pool names ...People are known to be abusing the lack of escaping in current libvirt to pass arbitrary args to QEMU.
I am one of those people--I have been storing colon-separated arguments in my disk source 'name' attribute to control the RBD cache settings on a per-volume basis. Under libvirt-1.0.6 such VM's refuse to start: error: Failed to start domain foo error: unsupported configuration: ':' not allowed in RBD source volume name 'mypool/foo:rbd_cache_size=268435456:rbd_cache_max_dirty=134217728:rbd_cache_target_dirty=33554432:rbd_cache_max_dirty_age=5'
So my question is how do I control the cache settings now?
You can set the cache mode with the cache attribute on the <driver> element inside a <disk> [1]. This works for rbd with QEMU 1.2 or later.
QEMU will try to read /etc/ceph/ceph.conf by default as well, so you can put more specific settings in a [global] or [client.foo] section there. Note that if you enable writeback caching here, you need to include cache='writeback' in the xml as well, or qemu will not send flushes.
Thanks for the quick reply. Is it safe to assume that the "foo" in "client.foo" should match the "auth username" attribute from the XML?
Yes, it should match the "auth usersname" attribute.
If I want different cache settings for different VM's or volumes on the same host is there a way to specify that any more? Can I specify multiple profiles in ceph.conf or point different qemu instances to different ceph.conf files?
Unfortunately there's no direct way anymore. You could use different rados clients and put custom settings in e.g. [client.x] and [client.y] sections of ceph.conf. That's really just a workaround though, since you probably don't want to create as many clients as you have sets of cache settings. There's no way to tell QEMU to read a non-default configuration file though, and libvirt runs QEMU in a clean environment so using CEPH_ARGS won't work either. Daniel and Eric, what do you think of exposing custom per-disk properties? QEMU did add support for driver-specific options recently. The rbd driver in QEMU could be modified to use those to receive ceph-specific options like cache size, and libvirt could provide a way to set them as opaque key/value pairs, i.e.: <disk> ... <source ...> <option name="foo" value="bar"/> </source> </disk> If you don't want this kind of general pass-through as part of the main api, maybe it could go in the qemu namespace? Josh
participants (2)
-
John Nielsen -
Josh Durgin