[libvirt] Libvirt domain event usage and consistency
Hi, I recently started to use the libvirt domain events. With them I increase the responsiveness of my VM state wachers. In general it works pretty well. I just listen to the events and do a periodic resync to cope with missed events. While watching the events I ran into a few interesting situations I wanted to share. The points 1-3 describe some minor issues or irregularities. Point 4 is about the fact that domain and state updates are not versioned which makes it very hard to stay in sync with libvirt when using events. My libvirt version is 1.2.18.4. 1) Event order seems to be weird on startup: When listening for VM lifecycle events I get this order: {"event_type": "Started", "timestamp": "2016-11-25T11:59:53.209326Z", "reason": "Booted", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"} {"event_type": "Defined", "timestamp": "2016-11-25T11:59:53.435530Z", "reason": "Added", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"} It is strange that a VM already boots before it is defined. Is this the intended order? 2) Defining a VM with VIR_DOMAIN_START_PAUSED gives me this event order {"event_type": "Defined", "timestamp": "2016-11-25T12:02:44.037817Z", "reason": "Added", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Resumed", "timestamp": "2016-11-25T12:02:44.813104Z", "reason": "Unpaused", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Started", "timestamp": "2016-11-25T12:02:44.813733Z", "reason": "Booted", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} This boot-order makes it hard to track active domains by listening to life-cycle events. One could theoretically still always fetch the VM state in the event callback and check the state, but if the state is not immediately transferred with the event itself, it can already be outdated, so this might be racy (intransparent for the libvirt bindings user), and as described in (3) currently not even possible. In general the real existing events seem to differ quite significantly from the described life-cycle in [1]. 3) "Defined" event is triggered before the domain is completely defined {"event_type": "Defined", "timestamp": "2016-11-25T12:02:44.037817Z", "reason": "Added", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Resumed", "timestamp": "2016-11-25T12:02:44.813104Z", "reason": "Unpaused", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Started", "timestamp": "2016-11-25T12:02:44.813733Z", "reason": "Booted", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} When I try to process the first event and do a xmldump I get: Event: [Code-42] [Domain-10] Domain not found: no domain with matching uuid 'b9906489-6d5b-40f8-a742-ca71b2b84277' (core_node) So it seems like I get the event before the domain is completely ready. 4) There libvirt domain description is not versioned I would expect that every time I update a domainxml (update from third party entity), or an event is generated (update from libvirt), that the resource version of a Domain is increased and that I get this resource version when I do a xmldump or when I get an event. Without this there is afaik no way to stay in sync with libvirt, even if you do regular polling of all domains. The main issue here is that I can never know if events in the queue arrived before my latest domain resync or after it. Also not that this is not about delivery guarantees of events. It is just about having a consistent view of a VM and the individual event. If I have resource versions, I can decide if an event is still interesting for me or not, which is exactly what I need to solve the syncing problem above. When I do a complete relisting of all domains to syn, I know which version I got and I can then see on every event if it is newer or older. If along side with the event, the domain xml, the VM state, and the resource version would be sent to a client, it would be even better. Then, whenever there is a new event for a VM in the queue, I can be sure that this domainxml I see is the one which triggered the event. This xml is then a complete representation for this revision number. Would be nice to hear your thoughts to these points. Best Regards, Roman [1] https://wiki.libvirt.org/page/VM_lifecycle#States_that_a_guest_domain_can_be...
On 25.11.2016 14:38, Roman Mohr wrote:
Hi,
I recently started to use the libvirt domain events. With them I increase the responsiveness of my VM state wachers. In general it works pretty well. I just listen to the events and do a periodic resync to cope with missed events.
While watching the events I ran into a few interesting situations I wanted to share. The points 1-3 describe some minor issues or irregularities. Point 4 is about the fact that domain and state updates are not versioned which makes it very hard to stay in sync with libvirt when using events.
My libvirt version is 1.2.18.4.
This might be the root cause. I'm unable to see some of the scenarios you're seeing. Have you tried the latest release (or even git HEAD) to check whether all the scenarios you are describing still stand?
1) Event order seems to be weird on startup:
When listening for VM lifecycle events I get this order:
{"event_type": "Started", "timestamp": "2016-11-25T11:59:53.209326Z", "reason": "Booted", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"} {"event_type": "Defined", "timestamp": "2016-11-25T11:59:53.435530Z", "reason": "Added", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"}
It is strange that a VM already boots before it is defined. Is this the intended order?
I don't see this order so probable this is fixed upstream.
2) Defining a VM with VIR_DOMAIN_START_PAUSED gives me this event order
I don't think you can define a domain with that flag. What's the actual action?
{"event_type": "Defined", "timestamp": "2016-11-25T12:02:44.037817Z", "reason": "Added", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Resumed", "timestamp": "2016-11-25T12:02:44.813104Z", "reason": "Unpaused", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Started", "timestamp": "2016-11-25T12:02:44.813733Z", "reason": "Booted", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"}
Interesting, so here is "defined" event delivered before the "started" event. Also - where is "suspended" event?
This boot-order makes it hard to track active domains by listening to life-cycle events. One could theoretically still always fetch the VM state in the event callback and check the state, but if the state is not immediately transferred with the event itself, it can already be outdated, so this might be racy (intransparent for the libvirt bindings user), and as described in (3) currently not even possible. In general the real existing events seem to differ quite significantly from the described life-cycle in [1].
Again, in the upstream I see something different: event 'lifecycle' for domain $domain: Started Booted event 'lifecycle' for domain $domain: Suspended Paused
3) "Defined" event is triggered before the domain is completely defined
{"event_type": "Defined", "timestamp": "2016-11-25T12:02:44.037817Z", "reason": "Added", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Resumed", "timestamp": "2016-11-25T12:02:44.813104Z", "reason": "Unpaused", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Started", "timestamp": "2016-11-25T12:02:44.813733Z", "reason": "Booted", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"}
When I try to process the first event and do a xmldump I get:
Event: [Code-42] [Domain-10] Domain not found: no domain with matching uuid 'b9906489-6d5b-40f8-a742-ca71b2b84277' (core_node)
So it seems like I get the event before the domain is completely ready.
You know that you shouldn't be calling libvirt APIs from event callbacks?
4) There libvirt domain description is not versioned
I would expect that every time I update a domainxml (update from third party entity), or an event is generated (update from libvirt), that the resource version of a Domain is increased and that I get this resource version when I do a xmldump or when I get an event. Without this there is afaik no way to stay in sync with libvirt, even if you do regular polling of all domains. The main issue here is that I can never know if events in the queue arrived before my latest domain resync or after it.
Also not that this is not about delivery guarantees of events. It is just about having a consistent view of a VM and the individual event. If I have resource versions, I can decide if an event is still interesting for me or not, which is exactly what I need to solve the syncing problem above. When I do a complete relisting of all domains to syn, I know which version I got and I can then see on every event if it is newer or older.
If along side with the event, the domain xml, the VM state, and the resource version would be sent to a client, it would be even better. Then, whenever there is a new event for a VM in the queue, I can be sure that this domainxml I see is the one which triggered the event. This xml is then a complete representation for this revision number.
I recall some people asking for this. Basically, they were worried about somebody from outside could manipulate their XMLs without them knowing. Frankly I don't recall what was our answer to that. Having a version number in live XML makes sense. However, it makes less sense for config XML - there would be no way how to start with version #0 once I've edited the file. Michal
On Fri, Nov 25, 2016 at 4:34 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 14:38, Roman Mohr wrote:
Hi,
I recently started to use the libvirt domain events. With them I increase the responsiveness of my VM state wachers. In general it works pretty well. I just listen to the events and do a periodic resync to cope with missed events.
While watching the events I ran into a few interesting situations I wanted to share. The points 1-3 describe some minor issues or irregularities. Point 4 is about the fact that domain and state updates are not versioned which makes it very hard to stay in sync with libvirt when using events.
My libvirt version is 1.2.18.4.
This might be the root cause. I'm unable to see some of the scenarios you're seeing. Have you tried the latest release (or even git HEAD) to check whether all the scenarios you are describing still stand?
Definitely better with latest HEAD but still it does not look completely right.
1) Event order seems to be weird on startup:
When listening for VM lifecycle events I get this order:
{"event_type": "Started", "timestamp": "2016-11-25T11:59:53.209326Z", "reason": "Booted", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"} {"event_type": "Defined", "timestamp": "2016-11-25T11:59:53.435530Z", "reason": "Added", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"}
It is strange that a VM already boots before it is defined. Is this the intended order?
I don't see this order so probable this is fixed upstream.
On latest master a normal creation emits these events: event 'lifecycle' for domain testvm: Resumed Unpaused event 'lifecycle' for domain testvm: Started Booted The Resumed event looks wrong. Further I get no more Defined/Undefined events. Maybe they were removed?
2) Defining a VM with VIR_DOMAIN_START_PAUSED gives me this event order
I don't think you can define a domain with that flag. What's the actual action?
That is the flag for the api, when using virsh using `--paused` does that.
{"event_type": "Defined", "timestamp": "2016-11-25T12:02:44.037817Z", "reason": "Added", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Resumed", "timestamp": "2016-11-25T12:02:44.813104Z", "reason": "Unpaused", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Started", "timestamp": "2016-11-25T12:02:44.813733Z", "reason": "Booted", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"}
Interesting, so here is "defined" event delivered before the "started" event. Also - where is "suspended" event?
With latest master the situation looks better. Now I see event 'lifecycle' for domain testvm: Started Booted event 'lifecycle' for domain testvm: Suspended Paused
This boot-order makes it hard to track active domains by listening to life-cycle events. One could theoretically still always fetch the VM
state
in the event callback and check the state, but if the state is not immediately transferred with the event itself, it can already be outdated, so this might be racy (intransparent for the libvirt bindings user), and as described in (3) currently not even possible. In general the real existing events seem to differ quite significantly from the described life-cycle in [1].
Again, in the upstream I see something different: event 'lifecycle' for domain $domain: Started Booted event 'lifecycle' for domain $domain: Suspended Paused
On master I see that too when I start the VM with `virsh create --paused`.
3) "Defined" event is triggered before the domain is completely defined
{"event_type": "Defined", "timestamp": "2016-11-25T12:02:44.037817Z", "reason": "Added", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Resumed", "timestamp": "2016-11-25T12:02:44.813104Z", "reason": "Unpaused", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Started", "timestamp": "2016-11-25T12:02:44.813733Z", "reason": "Booted", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"}
When I try to process the first event and do a xmldump I get:
Event: [Code-42] [Domain-10] Domain not found: no domain with matching uuid 'b9906489-6d5b-40f8-a742-ca71b2b84277' (core_node)
So it seems like I get the event before the domain is completely ready.
You know that you shouldn't be calling libvirt APIs from event callbacks?
No, good to know. Anyway, just tried to work around the above problems. So if the Defined/Undefined events were removed deliberately, then only the problem with the 'Resumed' event on a normal VM start remains.
4) There libvirt domain description is not versioned
I would expect that every time I update a domainxml (update from third party entity), or an event is generated (update from libvirt), that the resource version of a Domain is increased and that I get this resource version when I do a xmldump or when I get an event. Without this there is afaik no way to stay in sync with libvirt, even if you do regular polling of all domains. The main issue here is that I can never know if events in the queue arrived before my latest domain resync or after it.
Also not that this is not about delivery guarantees of events. It is just about having a consistent view of a VM and the individual event. If I
resource versions, I can decide if an event is still interesting for me or not, which is exactly what I need to solve the syncing problem above. When I do a complete relisting of all domains to syn, I know which version I got and I can then see on every event if it is newer or older.
If along side with the event, the domain xml, the VM state, and the resource version would be sent to a client, it would be even better. Then, whenever there is a new event for a VM in the queue, I can be sure that this domainxml I see is the one which triggered the event. This xml is
have then
a complete representation for this revision number.
I recall some people asking for this. Basically, they were worried about somebody from outside could manipulate their XMLs without them knowing. Frankly I don't recall what was our answer to that.
Having a version number in live XML makes sense. However, it makes less sense for config XML - there would be no way how to start with version #0 once I've edited the file.
Michal
On 25.11.2016 17:54, Roman Mohr wrote:
On Fri, Nov 25, 2016 at 4:34 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 14:38, Roman Mohr wrote:
Hi,
I recently started to use the libvirt domain events. With them I increase the responsiveness of my VM state wachers. In general it works pretty well. I just listen to the events and do a periodic resync to cope with missed events.
While watching the events I ran into a few interesting situations I wanted to share. The points 1-3 describe some minor issues or irregularities. Point 4 is about the fact that domain and state updates are not versioned which makes it very hard to stay in sync with libvirt when using events.
My libvirt version is 1.2.18.4.
This might be the root cause. I'm unable to see some of the scenarios you're seeing. Have you tried the latest release (or even git HEAD) to check whether all the scenarios you are describing still stand?
Definitely better with latest HEAD but still it does not look completely right.
1) Event order seems to be weird on startup:
When listening for VM lifecycle events I get this order:
{"event_type": "Started", "timestamp": "2016-11-25T11:59:53.209326Z", "reason": "Booted", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"} {"event_type": "Defined", "timestamp": "2016-11-25T11:59:53.435530Z", "reason": "Added", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"}
It is strange that a VM already boots before it is defined. Is this the intended order?
I don't see this order so probable this is fixed upstream.
On latest master a normal creation emits these events:
event 'lifecycle' for domain testvm: Resumed Unpaused event 'lifecycle' for domain testvm: Started Booted
The Resumed event looks wrong. Further I get no more Defined/Undefined events. Maybe they were removed?
Yes, they were removed. The Resumed event comes from qemu actually, because libvirt starts qemu in paused mode so that we can do some setup (e.g. place vcpu threads into cgroups) and only after that we can resume guest CPUs and in fact let guest start. Once this is done we deliberately emit Started event.
2) Defining a VM with VIR_DOMAIN_START_PAUSED gives me this event order
I don't think you can define a domain with that flag. What's the actual action?
That is the flag for the api, when using virsh using `--paused` does that.
Ah, that's for virsh create/start not virsh define. Anyway, this is no longer the case with upstream, is it?
{"event_type": "Defined", "timestamp": "2016-11-25T12:02:44.037817Z", "reason": "Added", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Resumed", "timestamp": "2016-11-25T12:02:44.813104Z", "reason": "Unpaused", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Started", "timestamp": "2016-11-25T12:02:44.813733Z", "reason": "Booted", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"}
Interesting, so here is "defined" event delivered before the "started" event. Also - where is "suspended" event?
With latest master the situation looks better. Now I see
event 'lifecycle' for domain testvm: Started Booted event 'lifecycle' for domain testvm: Suspended Paused
Again, both of these are deliberately emitted by libvirt and in fact I think they reflect what is happening. Michal
On Fri, Nov 25, 2016 at 6:04 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 17:54, Roman Mohr wrote:
On Fri, Nov 25, 2016 at 4:34 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 14:38, Roman Mohr wrote:
Hi,
I recently started to use the libvirt domain events. With them I increase the responsiveness of my VM state wachers. In general it works pretty well. I just listen to the events and do a periodic resync to cope with missed events.
While watching the events I ran into a few interesting situations I wanted to share. The points 1-3 describe some minor issues or irregularities. Point 4 is about the fact that domain and state updates are not versioned which makes it very hard to stay in sync with libvirt when using events.
My libvirt version is 1.2.18.4.
This might be the root cause. I'm unable to see some of the scenarios you're seeing. Have you tried the latest release (or even git HEAD) to check whether all the scenarios you are describing still stand?
Definitely better with latest HEAD but still it does not look completely right.
1) Event order seems to be weird on startup:
When listening for VM lifecycle events I get this order:
{"event_type": "Started", "timestamp": "2016-11-25T11:59:53.209326Z", "reason": "Booted", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"} {"event_type": "Defined", "timestamp": "2016-11-25T11:59:53.435530Z", "reason": "Added", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"}
It is strange that a VM already boots before it is defined. Is this the intended order?
I don't see this order so probable this is fixed upstream.
On latest master a normal creation emits these events:
event 'lifecycle' for domain testvm: Resumed Unpaused event 'lifecycle' for domain testvm: Started Booted
The Resumed event looks wrong. Further I get no more Defined/Undefined events. Maybe they were removed?
Yes, they were removed.
Nice
The Resumed event comes from qemu actually, because libvirt starts qemu in paused mode so that we can do some setup (e.g. place vcpu threads into cgroups) and only after that we can resume guest CPUs and in fact let guest start. Once this is done we deliberately emit Started event.
I would expect an event like this: event 'lifecycle' for domain testvm: Suspended Bootstrapping before the other two events. That takes the ambiguity from the Resumed event.
2) Defining a VM with VIR_DOMAIN_START_PAUSED gives me this event order
I don't think you can define a domain with that flag. What's the actual action?
That is the flag for the api, when using virsh using `--paused` does
that.
Ah, that's for virsh create/start not virsh define. Anyway, this is no longer the case with upstream, is it?
Right
{"event_type": "Defined", "timestamp": "2016-11-25T12:02:44.037817Z", "reason": "Added", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Resumed", "timestamp": "2016-11-25T12:02:44.813104Z", "reason": "Unpaused", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Started", "timestamp": "2016-11-25T12:02:44.813733Z", "reason": "Booted", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"}
Interesting, so here is "defined" event delivered before the "started" event. Also - where is "suspended" event?
With latest master the situation looks better. Now I see
event 'lifecycle' for domain testvm: Started Booted event 'lifecycle' for domain testvm: Suspended Paused
Again, both of these are deliberately emitted by libvirt and in fact I think they reflect what is happening.
Why is in this case not event 'lifecycle' for domain testvm: Resumed Unpaused event emitted? I would expect event 'lifecycle' for domain testvm: Resumed Unpaused event 'lifecycle' for domain testvm: Started Booted event 'lifecycle' for domain testvm: Suspended Paused So the situation on master is much better but because of the Resumed/Unpaused event I still have the feeling that the most simple but powerful usecase, watching for CREATE, UPDATE, DELETE is very hard because you can't know if the Resumed/Unpaused is the indicator for CREATE or UPDATE. What do you think? Michal
On 25.11.2016 18:33, Roman Mohr wrote:
On Fri, Nov 25, 2016 at 6:04 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 17:54, Roman Mohr wrote:
On Fri, Nov 25, 2016 at 4:34 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 14:38, Roman Mohr wrote:
Hi,
I recently started to use the libvirt domain events. With them I increase the responsiveness of my VM state wachers. In general it works pretty well. I just listen to the events and do a periodic resync to cope with missed events.
While watching the events I ran into a few interesting situations I wanted to share. The points 1-3 describe some minor issues or irregularities. Point 4 is about the fact that domain and state updates are not versioned which makes it very hard to stay in sync with libvirt when using events.
My libvirt version is 1.2.18.4.
This might be the root cause. I'm unable to see some of the scenarios you're seeing. Have you tried the latest release (or even git HEAD) to check whether all the scenarios you are describing still stand?
Definitely better with latest HEAD but still it does not look completely right.
1) Event order seems to be weird on startup:
When listening for VM lifecycle events I get this order:
{"event_type": "Started", "timestamp": "2016-11-25T11:59:53.209326Z", "reason": "Booted", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"} {"event_type": "Defined", "timestamp": "2016-11-25T11:59:53.435530Z", "reason": "Added", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"}
It is strange that a VM already boots before it is defined. Is this the intended order?
I don't see this order so probable this is fixed upstream.
On latest master a normal creation emits these events:
event 'lifecycle' for domain testvm: Resumed Unpaused event 'lifecycle' for domain testvm: Started Booted
The Resumed event looks wrong. Further I get no more Defined/Undefined events. Maybe they were removed?
Yes, they were removed.
Nice
The Resumed event comes from qemu actually, because libvirt starts qemu in paused mode so that we can do some setup (e.g. place vcpu threads into cgroups) and only after that we can resume guest CPUs and in fact let guest start. Once this is done we deliberately emit Started event.
I would expect an event like this:
event 'lifecycle' for domain testvm: Suspended Bootstrapping
before the other two events. That takes the ambiguity from the Resumed event.
2) Defining a VM with VIR_DOMAIN_START_PAUSED gives me this event order
I don't think you can define a domain with that flag. What's the actual action?
That is the flag for the api, when using virsh using `--paused` does
that.
Ah, that's for virsh create/start not virsh define. Anyway, this is no longer the case with upstream, is it?
Right
{"event_type": "Defined", "timestamp": "2016-11-25T12:02:44.037817Z", "reason": "Added", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Resumed", "timestamp": "2016-11-25T12:02:44.813104Z", "reason": "Unpaused", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Started", "timestamp": "2016-11-25T12:02:44.813733Z", "reason": "Booted", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"}
Interesting, so here is "defined" event delivered before the "started" event. Also - where is "suspended" event?
With latest master the situation looks better. Now I see
event 'lifecycle' for domain testvm: Started Booted event 'lifecycle' for domain testvm: Suspended Paused
Again, both of these are deliberately emitted by libvirt and in fact I think they reflect what is happening.
Why is in this case not
event 'lifecycle' for domain testvm: Resumed Unpaused
event emitted?
I would expect
event 'lifecycle' for domain testvm: Resumed Unpaused
I fail to see why this is supposed to be emitted. The QEMU is started as paused. If this even would be emitted, even if immediately followed by Paused event, it might suggest that guest CPUs have run for a little while. But that's not true.
event 'lifecycle' for domain testvm: Started Booted event 'lifecycle' for domain testvm: Suspended Paused
So the situation on master is much better but because of the Resumed/Unpaused event I still have the feeling that the most simple but powerful usecase, watching for CREATE, UPDATE, DELETE is very hard because you can't know if the Resumed/Unpaused is the indicator for CREATE or UPDATE.
It's neither. Resumed event reflects vCPU state, not the domain XML change. Maybe we need to document that better. For domain create event you need to watch for Started lifecycle event. For domain update event you need to watch for device-* event. For domain shutdown event you need to watch for Stopped lifecycle event. Michal
On Fri, Nov 25, 2016 at 8:23 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On Fri, Nov 25, 2016 at 6:04 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 17:54, Roman Mohr wrote:
On Fri, Nov 25, 2016 at 4:34 PM, Michal Privoznik <mprivozn@redhat.com
wrote:
On 25.11.2016 14:38, Roman Mohr wrote:
Hi,
I recently started to use the libvirt domain events. With them I increase the responsiveness of my VM state wachers. In general it works pretty well. I just listen to the events and do a periodic resync to cope with missed events.
While watching the events I ran into a few interesting situations I wanted to share. The points 1-3 describe some minor issues or irregularities. Point 4 is about the fact that domain and state updates are not versioned which makes it very hard to stay in sync with libvirt when using events.
My libvirt version is 1.2.18.4.
This might be the root cause. I'm unable to see some of the scenarios you're seeing. Have you tried the latest release (or even git HEAD) to check whether all the scenarios you are describing still stand?
Definitely better with latest HEAD but still it does not look completely right.
1) Event order seems to be weird on startup:
When listening for VM lifecycle events I get this order:
{"event_type": "Started", "timestamp": "2016-11-25T11:59:53.209326Z", "reason": "Booted", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"} {"event_type": "Defined", "timestamp": "2016-11-25T11:59:53.435530Z", "reason": "Added", "domain_name": "generic", "domain_id": "8ff7047b-fb46-44ff-a4c6-7c20c73ab86e"}
It is strange that a VM already boots before it is defined. Is this
On 25.11.2016 18:33, Roman Mohr wrote: the
intended order?
I don't see this order so probable this is fixed upstream.
On latest master a normal creation emits these events:
event 'lifecycle' for domain testvm: Resumed Unpaused event 'lifecycle' for domain testvm: Started Booted
The Resumed event looks wrong. Further I get no more Defined/Undefined events. Maybe they were removed?
Yes, they were removed.
Nice
The Resumed event comes from qemu actually, because libvirt starts qemu in paused mode so that we can do some setup (e.g. place vcpu threads into cgroups) and only after that we can resume guest CPUs and in fact let guest start. Once this is done we deliberately emit Started event.
I would expect an event like this:
event 'lifecycle' for domain testvm: Suspended Bootstrapping
before the other two events. That takes the ambiguity from the Resumed event.
2) Defining a VM with VIR_DOMAIN_START_PAUSED gives me this event
I don't think you can define a domain with that flag. What's the
actual
action?
That is the flag for the api, when using virsh using `--paused` does
order that.
Ah, that's for virsh create/start not virsh define. Anyway, this is no longer the case with upstream, is it?
Right
{"event_type": "Defined", "timestamp": "2016-11-25T12:02:44.037817Z", "reason": "Added", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Resumed", "timestamp": "2016-11-25T12:02:44.813104Z", "reason": "Unpaused", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"} {"event_type": "Started", "timestamp": "2016-11-25T12:02:44.813733Z", "reason": "Booted", "domain_name": "core_node", "domain_id": "b9906489-6d5b-40f8-a742-ca71b2b84277"}
Interesting, so here is "defined" event delivered before the "started" event. Also - where is "suspended" event?
With latest master the situation looks better. Now I see
event 'lifecycle' for domain testvm: Started Booted event 'lifecycle' for domain testvm: Suspended Paused
Again, both of these are deliberately emitted by libvirt and in fact I think they reflect what is happening.
Why is in this case not
event 'lifecycle' for domain testvm: Resumed Unpaused
event emitted?
I would expect
event 'lifecycle' for domain testvm: Resumed Unpaused
I fail to see why this is supposed to be emitted. The QEMU is started as paused. If this even would be emitted, even if immediately followed by Paused event, it might suggest that guest CPUs have run for a little while. But that's not true.
So to sum this up, to see if I get it right. When I directly start a VM I get: 1) event 'lifecycle' for domain testvm: Resumed Unpaused 2) event 'lifecycle' for domain testvm: Started Booted When I want to start it in paused mode I get: 1) event 'lifecycle' for domain testvm: Started Booted 2) event 'lifecycle' for domain testvm: Suspended Paused but no Resumed Unpaused as first event. The "Resumed Unpaused" I get in the first case, is because of internal libvirt VM setup on a paused VM, right? This work is not happening before the Started Booted event when I start the VM in paused mode?
event 'lifecycle' for domain testvm: Started Booted event 'lifecycle' for domain testvm: Suspended Paused
So the situation on master is much better but because of the Resumed/Unpaused event I still have the feeling that the most simple but powerful usecase, watching for CREATE, UPDATE, DELETE is very hard because you can't know if the Resumed/Unpaused is the indicator for CREATE or UPDATE.
It's neither. Resumed event reflects vCPU state, not the domain XML change. Maybe we need to document that better.
That would be great :)
For domain create event you need to watch for Started lifecycle event.
I have two questions here: 1) So the domain which triggers the resumed event does not exist before this Started even although I get notifications assigned to it? 2) Isn't the resumed event still ambiguous then? Normally it implies that the domain already exists and was paused, but in this case it reflects internal libvirt work on a not yet created domain. I don't think it is that much a deal, you can work around that, but it is something everyone has to keep in mind when working with these events. But even something simple like having a special event reason, let's say 'Resumed Initialized' would fix that, although the word 'Resumed' just sounds wrong in this context.
For domain update event you need to watch for device-* event. For domain shutdown event you need to watch for Stopped lifecycle event.
Michal
Roman
On Fri, Nov 25, 2016 at 4:34 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 14:38, Roman Mohr wrote:
[...]
4) There libvirt domain description is not versioned
I would expect that every time I update a domainxml (update from third party entity), or an event is generated (update from libvirt), that the resource version of a Domain is increased and that I get this resource version when I do a xmldump or when I get an event. Without this there is afaik no way to stay in sync with libvirt, even if you do regular polling of all domains. The main issue here is that I can never know if events in the queue arrived before my latest domain resync or after it.
Also not that this is not about delivery guarantees of events. It is just about having a consistent view of a VM and the individual event. If I
resource versions, I can decide if an event is still interesting for me or not, which is exactly what I need to solve the syncing problem above. When I do a complete relisting of all domains to syn, I know which version I got and I can then see on every event if it is newer or older.
If along side with the event, the domain xml, the VM state, and the resource version would be sent to a client, it would be even better. Then, whenever there is a new event for a VM in the queue, I can be sure that this domainxml I see is the one which triggered the event. This xml is
have then
a complete representation for this revision number.
I recall some people asking for this. Basically, they were worried about somebody from outside could manipulate their XMLs without them knowing. Frankly I don't recall what was our answer to that.
Having a version number in live XML makes sense. However, it makes less
sense for config XML - there would be no way how to start with version #0 once I've edited the file.
I think it would be very beneficial to have it on the config file too. Think about the resource version as opaque data which can be used by libvirt to see if the domain xml update contains the same resource number which libvirt sees. So if you want to be sure that you are updating the domain xml from the latest state, you pass in the resource version of your cached domain xml view. If the version is still the same inside of libvirt, libvirt updates the domain xml and increases the resource version. If it has changed in the meantime, it rejects the update and the client can re-fetch the latest state and try again. For classic update mode, just don't pass in the resource version as a client and libvirt can then just update the domain xml like always. This is pretty much the same principle like described in [1]. What is the rationale for this? I am mostly operating on cached views on libvirts data in combination with events. If, on listing resources and on events, I get a domain xml with a resource version and the Domain state, I have a full snapshot of the Domain, which I can put into a cache or queue. Then syncing with libvirt based on events and initial listing is possible. Otherwise I can never be sure if my view of libvirt is out of sync. When I then process an event I can process it based on the consistent snapshot view of the Domain and update the domain xml. If something has changed in the meantime, the update of the domain xml will fail and I can recheck and retry. Even better: In most cases the event does not need retries, because a newer event is already in the queue with the new Domain view which caused the update to fail. Finally it allows consistent incremental Domain state and description updates which can be sent to third parties without periodic refetching of all resources. Roman [1] https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventi...
Michal
On 06.12.2016 14:12, Roman Mohr wrote:
On Fri, Nov 25, 2016 at 4:34 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 14:38, Roman Mohr wrote:
[...]
4) There libvirt domain description is not versioned
I would expect that every time I update a domainxml (update from third party entity), or an event is generated (update from libvirt), that the resource version of a Domain is increased and that I get this resource version when I do a xmldump or when I get an event. Without this there is afaik no way to stay in sync with libvirt, even if you do regular polling of all domains. The main issue here is that I can never know if events in the queue arrived before my latest domain resync or after it.
Also not that this is not about delivery guarantees of events. It is just about having a consistent view of a VM and the individual event. If I
resource versions, I can decide if an event is still interesting for me or not, which is exactly what I need to solve the syncing problem above. When I do a complete relisting of all domains to syn, I know which version I got and I can then see on every event if it is newer or older.
If along side with the event, the domain xml, the VM state, and the resource version would be sent to a client, it would be even better. Then, whenever there is a new event for a VM in the queue, I can be sure that this domainxml I see is the one which triggered the event. This xml is
have then
a complete representation for this revision number.
I recall some people asking for this. Basically, they were worried about somebody from outside could manipulate their XMLs without them knowing. Frankly I don't recall what was our answer to that.
Having a version number in live XML makes sense. However, it makes less
sense for config XML - there would be no way how to start with version #0 once I've edited the file.
I think it would be very beneficial to have it on the config file too. Think about the resource version as opaque data which can be used by libvirt to see if the domain xml update contains the same resource number which libvirt sees.
If we do this then it's just a matter of time that users start demanding "How do I return to revision #1337?". And we don't want to go there.
So if you want to be sure that you are updating the domain xml from the latest state, you pass in the resource version of your cached domain xml view. If the version is still the same inside of libvirt, libvirt updates the domain xml and increases the resource version. If it has changed in the meantime, it rejects the update and the client can re-fetch the latest state and try again.
Makes sense.
For classic update mode, just don't pass in the resource version as a client and libvirt can then just update the domain xml like always. This is pretty much the same principle like described in [1].
What if users start mixing XMLs with and without revision IDs? Also, I'm a bit worried about revision ID wrap, but hopefully ULL is big enough.
What is the rationale for this?
I am mostly operating on cached views on libvirts data in combination with events. If, on listing resources and on events, I get a domain xml with a resource version and the Domain state, I have a full snapshot of the Domain, which I can put into a cache or queue. Then syncing with libvirt based on events and initial listing is possible. Otherwise I can never be sure if my view of libvirt is out of sync.
When I then process an event I can process it based on the consistent snapshot view of the Domain and update the domain xml. If something has changed in the meantime, the update of the domain xml will fail and I can recheck and retry. Even better: In most cases the event does not need retries, because a newer event is already in the queue with the new Domain view which caused the update to fail.
What if on event arrival you'd only note whatever actions you need to perform and only after you've processed the whole queue you start executing them? Michal
On Wed, Dec 7, 2016 at 8:26 AM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 06.12.2016 14:12, Roman Mohr wrote:
On Fri, Nov 25, 2016 at 4:34 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 14:38, Roman Mohr wrote:
[...]
4) There libvirt domain description is not versioned
I would expect that every time I update a domainxml (update from third party entity), or an event is generated (update from libvirt), that the resource version of a Domain is increased and that I get this resource version when I do a xmldump or when I get an event. Without this there is afaik no way to stay in sync with libvirt, even if you do regular polling of all domains. The main issue here is that I can never know if events in the queue arrived before my latest domain resync or after it.
Also not that this is not about delivery guarantees of events. It is just about having a consistent view of a VM and the individual event. If I
resource versions, I can decide if an event is still interesting for me or not, which is exactly what I need to solve the syncing problem above. When I do a complete relisting of all domains to syn, I know which version I got and I can then see on every event if it is newer or older.
If along side with the event, the domain xml, the VM state, and the resource version would be sent to a client, it would be even better. Then, whenever there is a new event for a VM in the queue, I can be sure that this domainxml I see is the one which triggered the event. This xml is
have then
a complete representation for this revision number.
I recall some people asking for this. Basically, they were worried about somebody from outside could manipulate their XMLs without them knowing. Frankly I don't recall what was our answer to that.
Having a version number in live XML makes sense. However, it makes less
sense for config XML - there would be no way how to start with version #0 once I've edited the file.
I think it would be very beneficial to have it on the config file too. Think about the resource version as opaque data which can be used by libvirt to see if the domain xml update contains the same resource number which libvirt sees.
If we do this then it's just a matter of time that users start demanding "How do I return to revision #1337?". And we don't want to go there.
Well - We could argue that currently it is just not possible to connect libvirtd events and a specific dom revision. This implies that users also could not do it themselves. With the change however, libvirtd will not provide the full history, but if it's needed then now users are able to add such functionality alongside of libvirtd. It would ismpliy enable this usecase.
So if you want to be sure that you are updating the domain xml from the latest state, you pass in the resource version of your cached domain xml view. If the version is still the same inside of libvirt, libvirt updates the domain xml and increases the resource version. If it has changed in the meantime, it rejects the update and the client can re-fetch the latest state and try again.
Makes sense.
For classic update mode, just don't pass in the resource version as a client and libvirt can then just update the domain xml like always. This is pretty much the same principle like described in [1].
What if users start mixing XMLs with and without revision IDs? Also, I'm a bit worried about revision ID wrap, but hopefully ULL is big enough.
I guess that mixing with and without IDs in subsequent updates should not be an issue. In the case with the ID it's a sanity check to ensure that the right state is getting updated. Without the ID this check is omitted. - fabian
What is the rationale for this?
I am mostly operating on cached views on libvirts data in combination with events. If, on listing resources and on events, I get a domain xml with a resource version and the Domain state, I have a full snapshot of the Domain, which I can put into a cache or queue. Then syncing with libvirt based on events and initial listing is possible. Otherwise I can never be sure if my view of libvirt is out of sync.
When I then process an event I can process it based on the consistent snapshot view of the Domain and update the domain xml. If something has changed in the meantime, the update of the domain xml will fail and I can recheck and retry. Even better: In most cases the event does not need retries, because a newer event is already in the queue with the new Domain view which caused the update to fail.
What if on event arrival you'd only note whatever actions you need to perform and only after you've processed the whole queue you start executing them?
Michal
-- Fabian Deutsch <fdeutsch@redhat.com> Associate Manager, RHV Hypervisor Red Hat
On Wed, Dec 7, 2016 at 8:38 AM, Fabian Deutsch <fdeutsch@redhat.com> wrote:
On 06.12.2016 14:12, Roman Mohr wrote:
On Fri, Nov 25, 2016 at 4:34 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 25.11.2016 14:38, Roman Mohr wrote:
[...]
4) There libvirt domain description is not versioned
I would expect that every time I update a domainxml (update from third party entity), or an event is generated (update from libvirt), that
resource version of a Domain is increased and that I get this resource version when I do a xmldump or when I get an event. Without this
afaik no way to stay in sync with libvirt, even if you do regular
of all domains. The main issue here is that I can never know if events in the queue arrived before my latest domain resync or after it.
Also not that this is not about delivery guarantees of events. It is just about having a consistent view of a VM and the individual event. If I have resource versions, I can decide if an event is still interesting for me or not, which is exactly what I need to solve the syncing problem above. When I do a complete relisting of all domains to syn, I know which version I got and I can then see on every event if it is newer or older.
If along side with the event, the domain xml, the VM state, and the resource version would be sent to a client, it would be even better. Then, whenever there is a new event for a VM in the queue, I can be sure
On Wed, Dec 7, 2016 at 8:26 AM, Michal Privoznik <mprivozn@redhat.com> wrote: the there is polling that
this domainxml I see is the one which triggered the event. This xml is then a complete representation for this revision number.
I recall some people asking for this. Basically, they were worried about somebody from outside could manipulate their XMLs without them knowing. Frankly I don't recall what was our answer to that.
Having a version number in live XML makes sense. However, it makes less
sense for config XML - there would be no way how to start with version #0 once I've edited the file.
I think it would be very beneficial to have it on the config file too. Think about the resource version as opaque data which can be used by libvirt to see if the domain xml update contains the same resource number which libvirt sees.
If we do this then it's just a matter of time that users start demanding "How do I return to revision #1337?". And we don't want to go there.
Well - We could argue that currently it is just not possible to connect libvirtd events and a specific dom revision. This implies that users also could not do it themselves. With the change however, libvirtd will not provide the full history, but if it's needed then now users are able to add such functionality alongside of libvirtd. It would ismpliy enable this usecase.
+1 If the dom and events are connected, creating your own history is just a matter of storing the dom in a map or a key-value store with the revision as a key. Really no need to have that in libvirt. In my eyes the benefit here really is that you get consistency which enables all the consistent caching and update usecases.
So if you want to be sure that you are updating the domain xml from the latest state, you pass in the resource version of your cached domain xml view. If the version is still the same inside of libvirt, libvirt updates the domain xml and increases the resource version. If it has changed in the meantime, it rejects the update and the client can re-fetch the latest state and try again.
Makes sense.
For classic update mode, just don't pass in the resource version as a client and libvirt can then just update the domain xml like always. This is pretty much the same principle like described in [1].
What if users start mixing XMLs with and without revision IDs? Also, I'm a bit worried about revision ID wrap, but hopefully ULL is big enough.
I guess that mixing with and without IDs in subsequent updates should not be an issue. In the case with the ID it's a sanity check to ensure that the right state is getting updated. Without the ID this check is omitted.
I can understand the concerns here. Maybe having a config value to enable resource versioning might make sense. Then we could enforce the presence of a resource version on updates. If versioning is turned off, everything is like always. This would problems like this completely.
- fabian
What is the rationale for this?
I am mostly operating on cached views on libvirts data in combination
with
events. If, on listing resources and on events, I get a domain xml with a resource version and the Domain state, I have a full snapshot of the Domain, which I can put into a cache or queue. Then syncing with libvirt based on events and initial listing is possible. Otherwise I can never be sure if my view of libvirt is out of sync.
When I then process an event I can process it based on the consistent snapshot view of the Domain and update the domain xml. If something has changed in the meantime, the update of the domain xml will fail and I can recheck and retry. Even better: In most cases the event does not need retries, because a newer event is already in the queue with the new Domain view which caused the update to fail.
What if on event arrival you'd only note whatever actions you need to perform and only after you've processed the whole queue you start executing them?
The problem I see here is that there can always be another event while I am executing an action, that makes it very hard to write well behaving code in combination with events.
Michal
-- Fabian Deutsch <fdeutsch@redhat.com> Associate Manager, RHV Hypervisor Red Hat
participants (3)
-
Fabian Deutsch -
Michal Privoznik -
Roman Mohr