[libvirt] Question about fsfreeze/fsthaw API
Hi all, Is there any plans to add APIs to execute fsfreeze/fsthaw in qemu guests? (something like virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags)) These would be useful in the case a guest has a disk device with its own snapshot feature, such as cinder volumes in OpenStack configuration. In such cases, libvirt clients want to issue fsfreeze/fsthaw before/after taking the disk snapshot. Currently we can execute them using virDomainQemuAgentCommand(). (e.g. virsh qemu-agent-command dom '{"execute":"guest-fsfreeze-freeze"}' ) However, this is exposing internal implementation too much. And it cannot leverage future implementation for the other hypervisors. So it would be nice if we have well-defined API for fsfreeze/fsthaw. If there is no plan for these API and this is acceptable, I will try to implement this. Any comments are welcome. Thanks, Tomoki Sekiyama
On 07.11.2013 01:31, Tomoki Sekiyama wrote:
Hi all,
Is there any plans to add APIs to execute fsfreeze/fsthaw in qemu guests? (something like virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags))
These would be useful in the case a guest has a disk device with its own snapshot feature, such as cinder volumes in OpenStack configuration. In such cases, libvirt clients want to issue fsfreeze/fsthaw before/after taking the disk snapshot.
Currently we can execute them using virDomainQemuAgentCommand(). (e.g. virsh qemu-agent-command dom '{"execute":"guest-fsfreeze-freeze"}' )
However, this is exposing internal implementation too much. And it cannot leverage future implementation for the other hypervisors. So it would be nice if we have well-defined API for fsfreeze/fsthaw.
If there is no plan for these API and this is acceptable, I will try to implement this. Any comments are welcome.
Thanks, Tomoki Sekiyama
I am not aware of somebody wanting these APIs exposed hence I don't know if somebody is working on it. But since there is a clear usage scenario I'd say go for it. Michal
On Thu, Nov 07, 2013 at 09:39:01AM +0100, Michal Privoznik wrote:
On 07.11.2013 01:31, Tomoki Sekiyama wrote:
Hi all,
Is there any plans to add APIs to execute fsfreeze/fsthaw in qemu guests? (something like virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags))
These would be useful in the case a guest has a disk device with its own snapshot feature, such as cinder volumes in OpenStack configuration. In such cases, libvirt clients want to issue fsfreeze/fsthaw before/after taking the disk snapshot.
Currently we can execute them using virDomainQemuAgentCommand(). (e.g. virsh qemu-agent-command dom '{"execute":"guest-fsfreeze-freeze"}' )
However, this is exposing internal implementation too much. And it cannot leverage future implementation for the other hypervisors. So it would be nice if we have well-defined API for fsfreeze/fsthaw.
If there is no plan for these API and this is acceptable, I will try to implement this. Any comments are welcome.
I am not aware of somebody wanting these APIs exposed hence I don't know if somebody is working on it. But since there is a clear usage scenario I'd say go for it.
We already support this QEMU agent commands in libvirt. We decided not to expose them explicitly as APIs, but instead wire them upto the impl of the disk snapshot APIs when VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE is used. Furthermore, OpenStack already has support for using this flag when doing snapshots, so I don't think we need any more work here. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 11/7/13 4:01 , "Daniel P. Berrange" <berrange@redhat.com> wrote:
On Thu, Nov 07, 2013 at 09:39:01AM +0100, Michal Privoznik wrote:
On 07.11.2013 01:31, Tomoki Sekiyama wrote:
Hi all,
Is there any plans to add APIs to execute fsfreeze/fsthaw in qemu guests? (something like virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags))
These would be useful in the case a guest has a disk device with its own snapshot feature, such as cinder volumes in OpenStack configuration. In such cases, libvirt clients want to issue fsfreeze/fsthaw before/after taking the disk snapshot.
Currently we can execute them using virDomainQemuAgentCommand(). (e.g. virsh qemu-agent-command dom '{"execute":"guest-fsfreeze-freeze"}' )
However, this is exposing internal implementation too much. And it cannot leverage future implementation for the other hypervisors. So it would be nice if we have well-defined API for fsfreeze/fsthaw.
If there is no plan for these API and this is acceptable, I will try to implement this. Any comments are welcome.
I am not aware of somebody wanting these APIs exposed hence I don't know if somebody is working on it. But since there is a clear usage scenario I'd say go for it.
We already support this QEMU agent commands in libvirt. We decided not to expose them explicitly as APIs, but instead wire them upto the impl of the disk snapshot APIs when VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE is used.
Furthermore, OpenStack already has support for using this flag when doing snapshots, so I don't think we need any more work here.
Daniel
AFAIK, VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE is used in openstack only when glusterfs is used as volume backend which uses external qcow2 snapshot. In the other cases, it uses own implementation instead of libvirt snapshot API, to exploit native snapshot features of various storages, such as LVM (over iSCSI), enterprise storages, and so on. For these cases, fsfreeze API (without taking snapshot) is useful. Tomoki Sekiyama
On Thu, Nov 07, 2013 at 03:45:56PM +0000, Tomoki Sekiyama wrote:
On 11/7/13 4:01 , "Daniel P. Berrange" <berrange@redhat.com> wrote:
On Thu, Nov 07, 2013 at 09:39:01AM +0100, Michal Privoznik wrote:
On 07.11.2013 01:31, Tomoki Sekiyama wrote:
Hi all,
Is there any plans to add APIs to execute fsfreeze/fsthaw in qemu guests? (something like virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags))
These would be useful in the case a guest has a disk device with its own snapshot feature, such as cinder volumes in OpenStack configuration. In such cases, libvirt clients want to issue fsfreeze/fsthaw before/after taking the disk snapshot.
Currently we can execute them using virDomainQemuAgentCommand(). (e.g. virsh qemu-agent-command dom '{"execute":"guest-fsfreeze-freeze"}' )
However, this is exposing internal implementation too much. And it cannot leverage future implementation for the other hypervisors. So it would be nice if we have well-defined API for fsfreeze/fsthaw.
If there is no plan for these API and this is acceptable, I will try to implement this. Any comments are welcome.
I am not aware of somebody wanting these APIs exposed hence I don't know if somebody is working on it. But since there is a clear usage scenario I'd say go for it.
We already support this QEMU agent commands in libvirt. We decided not to expose them explicitly as APIs, but instead wire them upto the impl of the disk snapshot APIs when VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE is used.
Furthermore, OpenStack already has support for using this flag when doing snapshots, so I don't think we need any more work here.
Daniel
AFAIK, VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE is used in openstack only when glusterfs is used as volume backend which uses external qcow2 snapshot.
In the other cases, it uses own implementation instead of libvirt snapshot API, to exploit native snapshot features of various storages, such as LVM (over iSCSI), enterprise storages, and so on. For these cases, fsfreeze API (without taking snapshot) is useful.
I'm not sure what your use case is, but I've been helping with getting blockdev-backup and drive-backup commands into qemu and hope to have some sort of libvirt API to expose them. These qemu commands allow you to create point-in-time snapshots of images with very low overhead regardless of underlying FS type etc. This was designed with 'image fleecing' in mind so you can examine the FS but it could also be used for backups etc. Ian
On 11/06/2013 05:31 PM, Tomoki Sekiyama wrote:
Hi all,
Is there any plans to add APIs to execute fsfreeze/fsthaw in qemu guests? (something like virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags))
I'm wondering if it might be better to have a single command with a callback argument, something like: virDomainQuiese(domain, timeout, callback, opaque, flags) which calls callback(domain, opaque) at the right point in time. I'm just a bit worried that since the freeze/thaw sequence is already handled as a pair by the QUIESCE flag of snapshot creation that exposing it as two non-atomic APIs may lead to inconsistent states that we'd have a hard time tracking which commands are allowed in which state. With only a single command and a callback, we have guaranteed semantics that all other API are locked out by our normal job mechanism, and that we can pair the freeze/thaw under the hood just as we do in snapshots. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Thu, Nov 7, 2013 at 11:39 AM, Eric Blake <eblake@redhat.com> wrote:
On 11/06/2013 05:31 PM, Tomoki Sekiyama wrote:
Hi all,
Is there any plans to add APIs to execute fsfreeze/fsthaw in qemu guests? (something like virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags))
I'm wondering if it might be better to have a single command with a callback argument, something like:
virDomainQuiese(domain, timeout, callback, opaque, flags)
which calls callback(domain, opaque) at the right point in time. I'm just a bit worried that since the freeze/thaw sequence is already handled as a pair by the QUIESCE flag of snapshot creation that exposing it as two non-atomic APIs may lead to inconsistent states that we'd have a hard time tracking which commands are allowed in which state. With only a single command and a callback, we have guaranteed semantics that all other API are locked out by our normal job mechanism, and that we can pair the freeze/thaw under the hood just as we do in snapshots.
I think this is a much better approach. We could arguably refactor the snapshot code to use this as well. -- Doug Goldstein
On 11/7/13 14:05 , "Doug Goldstein" <cardoe@gentoo.org> wrote:
On Thu, Nov 7, 2013 at 11:39 AM, Eric Blake <eblake@redhat.com> wrote:
On 11/06/2013 05:31 PM, Tomoki Sekiyama wrote:
Hi all,
Is there any plans to add APIs to execute fsfreeze/fsthaw in qemu guests? (something like virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags))
I'm wondering if it might be better to have a single command with a callback argument, something like:
virDomainQuiese(domain, timeout, callback, opaque, flags)
which calls callback(domain, opaque) at the right point in time. I'm just a bit worried that since the freeze/thaw sequence is already handled as a pair by the QUIESCE flag of snapshot creation that exposing it as two non-atomic APIs may lead to inconsistent states that we'd have a hard time tracking which commands are allowed in which state. With only a single command and a callback, we have guaranteed semantics that all other API are locked out by our normal job mechanism, and that we can pair the freeze/thaw under the hood just as we do in snapshots.
I think this is a much better approach. We could arguably refactor the snapshot code to use this as well.
This sounds reasonable for me. Thanks for your suggestion. Can we call virDomainBlockRebase() and BlockJobInfo()/BlockJobAbort() for the rebase operation in the callback? OpensStack is using these while image snapshot for now. For other kinds of volumes, other APIs might also be used. Tomoki Sekiyama
On Thu, Nov 07, 2013 at 10:39:08AM -0700, Eric Blake wrote:
On 11/06/2013 05:31 PM, Tomoki Sekiyama wrote:
Hi all,
Is there any plans to add APIs to execute fsfreeze/fsthaw in qemu guests? (something like virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags))
I'm wondering if it might be better to have a single command with a callback argument, something like:
virDomainQuiese(domain, timeout, callback, opaque, flags)
which calls callback(domain, opaque) at the right point in time. I'm just a bit worried that since the freeze/thaw sequence is already handled as a pair by the QUIESCE flag of snapshot creation that exposing it as two non-atomic APIs may lead to inconsistent states that we'd have a hard time tracking which commands are allowed in which state. With only a single command and a callback, we have guaranteed semantics that all other API are locked out by our normal job mechanism, and that we can pair the freeze/thaw under the hood just as we do in snapshots.
I don't see how 'callback' is going to work, since it would have to run client side, but everything else is server side. I'm not entirely comfortable with the callback idea in general for that - it seems it could create alot of "interesting" scenarios for us to worry about, in particular around re-entrancy & synchronization. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 11/8/13 0:25 , "Daniel P. Berrange" <berrange@redhat.com> wrote:
On Thu, Nov 07, 2013 at 10:39:08AM -0700, Eric Blake wrote:
On 11/06/2013 05:31 PM, Tomoki Sekiyama wrote:
Hi all,
Is there any plans to add APIs to execute fsfreeze/fsthaw in qemu guests? (something like virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags))
I'm wondering if it might be better to have a single command with a callback argument, something like:
virDomainQuiese(domain, timeout, callback, opaque, flags)
which calls callback(domain, opaque) at the right point in time. I'm just a bit worried that since the freeze/thaw sequence is already handled as a pair by the QUIESCE flag of snapshot creation that exposing it as two non-atomic APIs may lead to inconsistent states that we'd have a hard time tracking which commands are allowed in which state. With only a single command and a callback, we have guaranteed semantics that all other API are locked out by our normal job mechanism, and that we can pair the freeze/thaw under the hood just as we do in snapshots.
I don't see how 'callback' is going to work, since it would have to run client side, but everything else is server side. I'm not entirely comfortable with the callback idea in general for that - it seems it could create alot of "interesting" scenarios for us to worry about, in particular around re-entrancy & synchronization.
Thank you for the comments. I've investigated the current event mechanism of libvirt, and now thinking that it is difficult to be applied callback that runs in the client side.... It may require a new event mechanism that is sent only to the specific client, and that has notifier when callback is finished. However, it looks like overdesign for fsfreeze, so virDomainFSFreeze(domain,timeout,flags) and virDomainFSThaw(domain,timeout,flags) is much simpler. On Thu, Nov 07, 2013 at 10:39:08AM -0700, Eric Blake wrote:
With only a single command and a callback, we have guaranteed semantics that all other API are locked out by our normal job mechanism, and that we can pair the freeze/thaw under the hood just as we do in snapshots.
We should track fsfreeze status and fail the snapshot API with QUIESCE flag while it is frozen, instead of making this single APIs. Anyway, some block APIs is needed to be executable while guest filesystems are frozen to do a custom snapshot. Thanks, -- Tomoki Sekiyama
participants (6)
-
Daniel P. Berrange -
Doug Goldstein -
Eric Blake -
Ian Main -
Michal Privoznik -
Tomoki Sekiyama