4:28 a.m.
On 2015/3/28 18:06, Rui Chen wrote:
Thank you for reply, Chris.
2015-03-27 23:15 GMT+08:00 Chris Friesen <chris.friesen(a)windriver.com
<mailto:chris.friesen@windriver.com>>:
On 03/26/2015 07:44 PM, Rui Chen wrote:
Yes, you are right, but we found our instance hang at first dom.shutdown() call,
if the dom.shutdown() don't return, there is no chance to execute
dom.destroy(),
right?
Correct. The code is written assuming dom.shutdown() cannot block indefinitely.
The libvirt docs at
"https://libvirt.org/html/__libvirt-libvirt-domain.html#__virDomainShutdown
<https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainShutdown>... say
"...this command returns as soon as the shutdown request is issued rather than
blocking until the guest is no longer running."
If dom.shutdown() blocks indefinitely, then that's a libvirt bug.
Chris
The API virDomainShutdown's description is out of date, it's not correct.
In fact, virDomainShutdown would block or not, depending on its mode. If it's in
mode *agent*, then it would be blocked until qemu founds that the guest actually got
down.
Otherwise, if it's in mode *acpi*, then it would return immediately.
Thus, maybe further more work need to be done in Openstack.
What's your opinions, Michal and Daniel (from libvirt.org), and Chris (from
openstack.org) :)
7:08 a.m.
New subject: [libvirt] [openstack-dev] [nova] The risk of hanging when shutdown instance.
On 30.03.2015 11:28, zhang bo wrote:
On 2015/3/28 18:06, Rui Chen wrote:
> <snip/>
The API virDomainShutdown's description is out of date, it's not correct.
In fact, virDomainShutdown would block or not, depending on its mode. If it's in
mode *agent*, then it would be blocked until qemu founds that the guest actually got
down.
Otherwise, if it's in mode *acpi*, then it would return immediately.
Thus, maybe further more work need to be done in Openstack.
What's your opinions, Michal and Daniel (from libvirt.org), and Chris (from
openstack.org) :)
Yep, the documentation could be better in that respect. I've proposed a
patch on the libvirt upstream list:
https://www.redhat.com/archives/libvir-list/2015-March/msg01533.html
Michal
3:36 p.m.
New subject: [libvirt] [openstack-dev] [nova] The risk of hanging when shutdown instance.
On 03/30/2015 06:08 AM, Michal Privoznik wrote:
On 30.03.2015 11:28, zhang bo wrote:
> On 2015/3/28 18:06, Rui Chen wrote:
>
>> <snip/>
>
> The API virDomainShutdown's description is out of date, it's not correct.
> In fact, virDomainShutdown would block or not, depending on its mode. If it's
in mode *agent*, then it would be blocked until qemu founds that the guest actually got
down.
> Otherwise, if it's in mode *acpi*, then it would return immediately.
> Thus, maybe further more work need to be done in Openstack.
>
> What's your opinions, Michal and Daniel (from libvirt.org), and Chris (from
openstack.org) :)
>
Yep, the documentation could be better in that respect. I've proposed a
patch on the libvirt upstream list:
https://www.redhat.com/archives/libvir-list/2015-March/msg01533.html
I don't think a doc patch is right. If you don't pass any flags, then
it is up to the hypervisor which method it will attempt (agent or ACPI).
Yes, explicitly requesting an agent as the only method to attempt might
be justifiable as a reason to block, but the overall API contract is to
NOT block indefinitely. I think that rather than a doc patch, we need
to fix the underlying bug, and guarantee that we return after a finite
time even when the agent is involved.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
10:37 p.m.
New subject: [libvirt] [openstack-dev] [nova] The risk of hanging when shutdown instance.
On 2015/3/31 4:36, Eric Blake wrote:
On 03/30/2015 06:08 AM, Michal Privoznik wrote:
> On 30.03.2015 11:28, zhang bo wrote:
>> On 2015/3/28 18:06, Rui Chen wrote:
>>
>>> <snip/>
>>
>> The API virDomainShutdown's description is out of date, it's not
correct.
>> In fact, virDomainShutdown would block or not, depending on its mode. If
it's in mode *agent*, then it would be blocked until qemu founds that the guest
actually got down.
>> Otherwise, if it's in mode *acpi*, then it would return immediately.
>> Thus, maybe further more work need to be done in Openstack.
>>
>> What's your opinions, Michal and Daniel (from libvirt.org), and Chris (from
openstack.org) :)
>>
>
>
> Yep, the documentation could be better in that respect. I've proposed a
> patch on the libvirt upstream list:
>
> https://www.redhat.com/archives/libvir-list/2015-March/msg01533.html
I don't think a doc patch is right. If you don't pass any flags, then
it is up to the hypervisor which method it will attempt (agent or ACPI).
Yes, explicitly requesting an agent as the only method to attempt might
be justifiable as a reason to block, but the overall API contract is to
NOT block indefinitely. I think that rather than a doc patch, we need
to fix the underlying bug, and guarantee that we return after a finite
time even when the agent is involved.
So, may we get to a final decision? :) Shall we timeout in virDomainShutdown() or leave it
to openstack?
The 2 solutions I can see are:
1) timeout in virDomainShutdown() and virDomainReboot(). in libvirt.
2) spawn a new thread to monitor the guest's status, if it's not shutoff after
dom.shutdown() for a while,
call dom.destroy() to force shut it down. in openstack.
8:55 a.m.
New subject: [libvirt] [openstack-dev] [nova] The risk of hanging when shutdown instance.
On Tue, Mar 31, 2015 at 11:37:04AM +0800, zhang bo wrote:
On 2015/3/31 4:36, Eric Blake wrote:
> On 03/30/2015 06:08 AM, Michal Privoznik wrote:
>> On 30.03.2015 11:28, zhang bo wrote:
>>> On 2015/3/28 18:06, Rui Chen wrote:
>>>
>>>> <snip/>
>>>
>>> The API virDomainShutdown's description is out of date, it's not
correct.
>>> In fact, virDomainShutdown would block or not, depending on its mode. If
it's in mode *agent*, then it would be blocked until qemu founds that the guest
actually got down.
>>> Otherwise, if it's in mode *acpi*, then it would return immediately.
>>> Thus, maybe further more work need to be done in Openstack.
>>>
>>> What's your opinions, Michal and Daniel (from libvirt.org), and Chris
(from openstack.org) :)
>>>
>>
>>
>> Yep, the documentation could be better in that respect. I've proposed a
>> patch on the libvirt upstream list:
>>
>> https://www.redhat.com/archives/libvir-list/2015-March/msg01533.html
>
> I don't think a doc patch is right. If you don't pass any flags, then
> it is up to the hypervisor which method it will attempt (agent or ACPI).
> Yes, explicitly requesting an agent as the only method to attempt might
> be justifiable as a reason to block, but the overall API contract is to
> NOT block indefinitely. I think that rather than a doc patch, we need
> to fix the underlying bug, and guarantee that we return after a finite
> time even when the agent is involved.
>
So, may we get to a final decision? :) Shall we timeout in virDomainShutdown() or leave
it to openstack?
The 2 solutions I can see are:
1) timeout in virDomainShutdown() and virDomainReboot(). in libvirt.
2) spawn a new thread to monitor the guest's status, if it's not shutoff after
dom.shutdown() for a while,
call dom.destroy() to force shut it down. in openstack.
We should probably do both.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
9:04 a.m.
New subject: [libvirt] [openstack-dev] [nova] The risk of hanging when shutdown instance.
On 03/30/2015 11:37 PM, zhang bo wrote:
On 2015/3/31 4:36, Eric Blake wrote:
> On 03/30/2015 06:08 AM, Michal Privoznik wrote:
>> On 30.03.2015 11:28, zhang bo wrote:
>>> On 2015/3/28 18:06, Rui Chen wrote:
>>>
>>>> <snip/>
>>> The API virDomainShutdown's description is out of date, it's not
correct.
>>> In fact, virDomainShutdown would block or not, depending on its mode. If
it's in mode *agent*, then it would be blocked until qemu founds that the guest
actually got down.
>>> Otherwise, if it's in mode *acpi*, then it would return immediately.
>>> Thus, maybe further more work need to be done in Openstack.
>>>
>>> What's your opinions, Michal and Daniel (from libvirt.org), and Chris
(from openstack.org) :)
>>>
>>
>> Yep, the documentation could be better in that respect. I've proposed a
>> patch on the libvirt upstream list:
>>
>> https://www.redhat.com/archives/libvir-list/2015-March/msg01533.html
> I don't think a doc patch is right. If you don't pass any flags, then
> it is up to the hypervisor which method it will attempt (agent or ACPI).
> Yes, explicitly requesting an agent as the only method to attempt might
> be justifiable as a reason to block, but the overall API contract is to
> NOT block indefinitely. I think that rather than a doc patch, we need
> to fix the underlying bug, and guarantee that we return after a finite
> time even when the agent is involved.
>
So, may we get to a final decision? :) Shall we timeout in virDomainShutdown() or leave
it to openstack?
The 2 solutions I can see are:
1) timeout in virDomainShutdown() and virDomainReboot(). in libvirt.
2) spawn a new thread to monitor the guest's status, if it's not shutoff after
dom.shutdown() for a while,
call dom.destroy() to force shut it down. in openstack.
And to complicate things a bit further, when you call dom.destroy() you
should probably first call it with the VIR_DOMAIN_DESTROY_GRACEFUL flag
(which will send SIGTERM to qemu, but never SIGKILL), so that qemu can
get a chance to flush the disk image files to disk, then if that fails,
call dom.destroy() again without that flag (which sends SIGTERM to qemu,
and if that fails after awhile it will send SIGKILL)
Here's the email for the patch that added DESTROY_GRACEFUL for a better
explanation:
http://www.redhat.com/archives/libvir-list/2012-February/msg00124.html
3544
days inactive
3555
days old
5 comments
5 participants
participants (5)
-
Daniel P. Berrange -
Eric Blake -
Laine Stump -
Michal Privoznik -
zhang bo