5:33 p.m.
[re-adding the list]
On 08/01/2011 04:22 PM, Jonathan Stoppani wrote:
On Aug 1, 2011, at 16:20 , Eric Blake wrote:
> On 08/01/2011 04:11 PM, Jonathan Stoppani wrote:
>> Hi there,
>>
>> I'm trying to migrate a domain between two Gentoo hosts using KVM as
hypervisor, but the migration hangs.
>> I tried both live or offline migration modes without success.
>
> Does your 'nc' command have a -q option? If so, then this is probably
>
> https://bugzilla.redhat.com/show_bug.cgi?id=726174
>
Thanks for the prompt answer Eric! Yes, nc has a q option:
-q, --hold-timeout=SEC1[:SEC2] Set hold timeout(s) for local [and remote]
Glad to hear that we found root cause to your problems, then.
The bug specifically refers to ssh, does that mean that it should work over tcp?
The problem is that libvirt is trying to start a remote nc session over
ssh; but looking at http://libvirt.org/remote.html, it looks like ssh is
the only protocol using nc in that manner (so yes, you can probably
avoid the issue by using tcp or tls). Meanwhile, I think you can work
around it without patching libvirt, by using this as your remote URI:
qemu+ssh://user@remotehost/system?netcat=/path/to/nc-wrapper
where nc-wrapper is an executable script installed on remotehost,
looking like:
#!/bin/sh
exec /path/to/real/nc -q0 "$@"
--
Eric Blake eblake(a)redhat.com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
5:50 p.m.
New subject: [libvirt-users] Migration hangs on Gentoo with KVM
On Aug 1, 2011, at 16:33 , Eric Blake wrote:
[re-adding the list]
Sorry about that, still not used to mailman lists which don't put the list address in
the reply-to field. ;-)
> Thanks for the prompt answer Eric! Yes, nc has a q option:
>
> -q, --hold-timeout=SEC1[:SEC2] Set hold timeout(s) for local [and remote]
Glad to hear that we found root cause to your problems, then.
>
> The bug specifically refers to ssh, does that mean that it should work over tcp?
The problem is that libvirt is trying to start a remote nc session over ssh; but looking
at http://libvirt.org/remote.html, it looks like ssh is the only protocol using nc in that
manner (so yes, you can probably avoid the issue by using tcp or tls). Meanwhile, I think
you can work around it without patching libvirt, by using this as your remote URI:
qemu+ssh://user@remotehost/system?netcat=/path/to/nc-wrapper
where nc-wrapper is an executable script installed on remotehost, looking like:
#!/bin/sh
exec /path/to/real/nc -q0 "$@"
Just tried this, but still hangs; will try tcp and report the results.
~Jonathan
7:01 p.m.
New subject: [libvirt-users] Migration hangs on Gentoo with KVM
On Aug 1, 2011, at 16:50 , Jonathan Stoppani wrote:
On Aug 1, 2011, at 16:33 , Eric Blake wrote:
> [re-adding the list]
Sorry about that, still not used to mailman lists which don't put the list address in
the reply-to field. ;-)
>> Thanks for the prompt answer Eric! Yes, nc has a q option:
>>
>> -q, --hold-timeout=SEC1[:SEC2] Set hold timeout(s) for local [and remote]
>
> Glad to hear that we found root cause to your problems, then.
>
>>
>> The bug specifically refers to ssh, does that mean that it should work over tcp?
>
> The problem is that libvirt is trying to start a remote nc session over ssh; but
looking at http://libvirt.org/remote.html, it looks like ssh is the only protocol using nc
in that manner (so yes, you can probably avoid the issue by using tcp or tls). Meanwhile,
I think you can work around it without patching libvirt, by using this as your remote
URI:
>
> qemu+ssh://user@remotehost/system?netcat=/path/to/nc-wrapper
>
> where nc-wrapper is an executable script installed on remotehost, looking like:
>
> #!/bin/sh
> exec /path/to/real/nc -q0 "$@"
Just tried this, but still hangs; will try tcp and report the results.
~Jonathan
Tested using qemu+tcp and it hangs the same. If I interrupt the migration (^C), the domain
is correctly destroyed on the destination but left in the paused state on the source. If I
try to start it manually, I obtain this error:
# virsh resume 1
error: Failed to resume domain 1
error: Timed out during operation: cannot acquire state change lock
Any insights?
~Jonathan
3:30 p.m.
New subject: [libvirt-users] Migration hangs on Gentoo with KVM
On Aug 1, 2011, at 18:01 , Jonathan Stoppani wrote:
On Aug 1, 2011, at 16:50 , Jonathan Stoppani wrote:
>
> On Aug 1, 2011, at 16:33 , Eric Blake wrote:
>
>> [re-adding the list]
>
> Sorry about that, still not used to mailman lists which don't put the list
address in the reply-to field. ;-)
>
>>> Thanks for the prompt answer Eric! Yes, nc has a q option:
>>>
>>> -q, --hold-timeout=SEC1[:SEC2] Set hold timeout(s) for local [and remote]
>>
>> Glad to hear that we found root cause to your problems, then.
>>
>>>
>>> The bug specifically refers to ssh, does that mean that it should work over
tcp?
>>
>> The problem is that libvirt is trying to start a remote nc session over ssh; but
looking at http://libvirt.org/remote.html, it looks like ssh is the only protocol using nc
in that manner (so yes, you can probably avoid the issue by using tcp or tls). Meanwhile,
I think you can work around it without patching libvirt, by using this as your remote
URI:
>>
>> qemu+ssh://user@remotehost/system?netcat=/path/to/nc-wrapper
>>
>> where nc-wrapper is an executable script installed on remotehost, looking like:
>>
>> #!/bin/sh
>> exec /path/to/real/nc -q0 "$@"
>
> Just tried this, but still hangs; will try tcp and report the results.
>
> ~Jonathan
Tested using qemu+tcp and it hangs the same. If I interrupt the migration (^C), the
domain is correctly destroyed on the destination but left in the paused state on the
source. If I try to start it manually, I obtain this error:
# virsh resume 1
error: Failed to resume domain 1
error: Timed out during operation: cannot acquire state change lock
Any insights?
Can someone shed some light on the libvirt locking possibilities? It seems to me that
sanlock is not supported on gentoo (and libvirt is compiled using --without-sanlock);
could this be the cause of the problem?
Is there some way to explicitly set the locking mechanism to a noop in the libvirt
configuration?
~Jonathan
3:45 p.m.
New subject: [libvirt-users] Migration hangs on Gentoo with KVM
On 08/17/2011 02:30 PM, Jonathan Stoppani wrote:
>>>> Thanks for the prompt answer Eric! Yes, nc has a q
option:
>>>>
>>>> -q, --hold-timeout=SEC1[:SEC2] Set hold timeout(s) for local [and
remote]
>>>
We still haven't incorporated patches to autodetect nc usage on the
remote side (some have been proposed by Guido, but there were some
additional issues to address first). Hopefully by 0.9.5...
Until that is fixed, then it very well could be that you are deadlocking
the libvirtd handling of the remote connection due to nc holding the
connection open too long, explaining while all further attempts to do
something with the domain are getting stuck waiting for the nc
connection to resolve.
> Tested using qemu+tcp and it hangs the same. If I interrupt the
migration (^C), the domain is correctly destroyed on the destination but left in the
paused state on the source. If I try to start it manually, I obtain this error:
>
> # virsh resume 1
> error: Failed to resume domain 1
> error: Timed out during operation: cannot acquire state change lock
This is the internal mutex lock used for serializing access to libvirt
internal structures, such as when coordinating with a remote server
(which coordination involves the use of nc). When you get this message,
about the only thing you can do is restart libvirtd. Which version of
libvirt were you testing? 0.9.4 adds quite a few improvements on being
able to gracefully recover from failed migrations.
>
> Any insights?
Can someone shed some light on the libvirt locking possibilities? It seems to me that
sanlock is not supported on gentoo (and libvirt is compiled using --without-sanlock);
could this be the cause of the problem?
Completely unrelated. sanlock is a program for controlling access to
shared file storage, and has nothing to do with the internal mutex lock
failure message you quoted above.
Is there some way to explicitly set the locking mechanism to a noop
in the libvirt configuration?
You are confusing two terms; using the sanlock or no-op disk manager has
nothing to do with libvirtd getting confused and deadlocking on internal
data structures. If you built --without-sanlock, then you are already
using the no-op disk manager; but if sanlock is compiled in, you control
whether to use it by modifying /etc/libvirt/qemu.conf. But making a
configuration change there won't affect the problem you actually saw above.
--
Eric Blake eblake(a)redhat.com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
4846
days inactive
4862
days old
4 comments
2 participants
participants (2)
-
Eric Blake -
Jonathan Stoppani