[libvirt-users] P2P live migration with non-shared storage: fails to connect to remote libvirt URI qemu+ssh
3:08 a.m.
Migration without --p2p works just fine, ie. the below works:
$ virsh migrate --verbose --copy-storage-all \
--live cvm1 qemu+ssh://kashyapc@devstack3/system
Migration: [100 %]
Result:
- On the source host, the guest is shut off
- On the destination host, the guest is live migratied successfully
Migration with "--p2p" fails, a simple test below:
First, I should note, I didn't modify any settings in
/etc/libvirt/libvirtd.conf on both source and destination hosts, except for
libvirt logging filters.
(0) On source and destination hosts, SSH keys are setup so that passwordless
auth works:
$ ssh-keygen -t
$ eval `ssh-agent`
$ ssh-add .ssh/id_rsa
$ ssh-copy-id root@devstack3
(1) Check if the connection to the remote host works w/o a prompt for
user credentials (the below works as user and root):
$ virsh -c qemu+ssh://kashyapc@devstack3/system
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh #
(2) Perform peer to peer live migration (as root):
$ virsh migrate --verbose --p2p --copy-storage-all \
--live cvm1 qemu+ssh://kashyapc@devstack3/system
error: operation failed: Failed to connect to remote libvirt URI
qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied, please try
again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection reset
by peer
From libvirtd debug log:
[. . .]
2015-04-03 06:04:16.221+0000: 31009: debug : virCommandRunAsync:2408 : About to run
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin ssh -l kashyapc devstack3
sh -c ''\''if '\
''nc'\'' -q 2>&1 | grep "requires an argument"
>/dev/null 2>&1; then ARG=-q0;else ARG=;fi;'\''nc'\''
$ARG -U /var/run/libvirt/libvirt-sock'\'''
2015-04-03 06:04:16.223+0000: 31009: debug : virCommandRunAsync:2411 : Command result 0,
with PID 11204
2015-04-03 06:04:16.300+0000: 31009: error : virNetSocketReadWire:1564 : Cannot recv data:
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection reset by
peer
2015-04-03 06:04:16.300+0000: 31009: debug : do_open:1194 : driver 6 remote returned
ERROR
2015-04-03 06:04:16.300+0000: 31009: debug : qemuDomainObjExitRemote:1695 : Exited remote
(vm=0x7f727c005f80 name=cvm1)
2015-04-03 06:04:16.300+0000: 31009: error : doPeer2PeerMigrate:4711 : operation failed:
Failed to connect to remote libvirt URI qemu+ssh://kashyapc@devstack3/system: Cannot recv
data: Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection reset by
peer
2015-04-03 06:04:16.300+0000: 31009: debug : qemuMigrationRestoreDomainState:1429 :
driver=0x7f728c160980, vm=0x7f727c005f80, pre-mig-state=1, state=1
2015-04-03 06:04:16.300+0000: 31009: debug : qemuDomainObjEndAsyncJob:1497 : Stopping
async job: migration out (vm=0x7f727c005f80 name=cvm1)
2015-04-03 06:04:16.301+0000: 31007: debug : virProcessAbort:167 : aborting child process
11204
2015-04-03 06:04:16.301+0000: 31007: debug : virProcessAbort:175 : trying SIGTERM to child
process 11204
[. . .]
What else am I missing?
--
/kashyap
3:13 a.m.
On Fri, Apr 03, 2015 at 10:08:21AM +0200, Kashyap Chamarthy wrote:
Migration without --p2p works just fine, ie. the below works:
$ virsh migrate --verbose --copy-storage-all \
--live cvm1 qemu+ssh://kashyapc@devstack3/system
Migration: [100 %]
[. . .]
(2) Perform peer to peer live migration (as root):
$ virsh migrate --verbose --p2p --copy-storage-all \
--live cvm1 qemu+ssh://kashyapc@devstack3/system
error: operation failed: Failed to connect to remote libvirt URI
qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied, please try
again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection
reset by peer
Just to note, both the tests were performed with the below versions on
source and destination hosts:
kernel-4.0.0-0.rc5.git4.1.fc22.x86_64
libvirt-daemon-kvm-1.2.13-2.fc22.x86_64
qemu-system-x86-2.3.0-0.2.rc1.fc22.x86_64
[. . .]
--
/kashyap
5:27 a.m.
On Fri, Apr 03, 2015 at 10:13:38AM +0200, Kashyap Chamarthy wrote:
On Fri, Apr 03, 2015 at 10:08:21AM +0200, Kashyap Chamarthy wrote:
> Migration without --p2p works just fine, ie. the below works:
>
>
> $ virsh migrate --verbose --copy-storage-all \
> --live cvm1 qemu+ssh://kashyapc@devstack3/system
> Migration: [100 %]
>
[. . .]
>
> (2) Perform peer to peer live migration (as root):
>
> $ virsh migrate --verbose --p2p --copy-storage-all \
> --live cvm1 qemu+ssh://kashyapc@devstack3/system
Hmm, reading the flow for peer2peer here:
https://libvirt.org/migration.html#flowpeer2peer
which says:
"Note that the source libvirtd uses its own credentials (typically root)
to connect to the destination, rather than the credentials used by the
client to connect to the source; if these differ, it is common to run
into a situation where a client can connect to the destination directly
but the source cannot make the connection to set up the peer-to-peer
migration."
So, somehow libvirtd daemon on source is not able to pick the
credentials. I'll try qemu+tcp with the below configs in libvirtd.conf
(note: the auth_tcp = "none" because, I'm doing the test in a trusted
network).
listen_tls = 0
listen_tcp = 1
auth_tcp = "none"
--
/kashyap
2:17 p.m.
New subject: [libvirt-users] [RESOLVED] Re: P2P live migration with non-shared storage: fails to connect to remote libvirt URI qemu+ssh
On Fri, Apr 03, 2015 at 12:27:07PM +0200, Kashyap Chamarthy wrote:
On Fri, Apr 03, 2015 at 10:13:38AM +0200, Kashyap Chamarthy wrote:
> On Fri, Apr 03, 2015 at 10:08:21AM +0200, Kashyap Chamarthy wrote:
> > Migration without --p2p works just fine, ie. the below works:
> >
> >
> > $ virsh migrate --verbose --copy-storage-all \
> > --live cvm1 qemu+ssh://kashyapc@devstack3/system
> > Migration: [100 %]
Problem found:
I was running the `ssh-agent` with a passphrase for the SSH key. This
will work in the managed direct migration case as above (because the
client handles process controls most of it). . .
>
> [. . .]
>
> >
> > (2) Perform peer to peer live migration (as root):
> >
> > $ virsh migrate --verbose --p2p --copy-storage-all \
> > --live cvm1 qemu+ssh://kashyapc@devstack3/system
. . .but _not_ in P2P case, because the libvirt daemon on the source
_does not_ have access to the `ssh-agent` (so while I _can_ SSH into the
dest host as 'root' w/o password, libvirt daemon cannot! So ensure that
SSH key has no pass-phrase (only in trusted networks for testing only!)
or setup X.509 certificates.
When Jirka asked me on IRC that if I was able to SSH as root to the
destination host, I was able to, but didn't think of the `ssh-agent`
holding the private key _with_ passphrase!
Hmm, reading the flow for peer2peer here:
https://libvirt.org/migration.html#flowpeer2peer
which says:
"Note that the source libvirtd uses its own credentials (typically root)
to connect to the destination, rather than the credentials used by the
client to connect to the source; if these differ, it is common to run
into a situation where a client can connect to the destination directly
but the source cannot make the connection to set up the peer-to-peer
migration."
Also thanks to Lars Kellogg-Stedman (OpenStack dev) who reminded me of
the the question "does your libvirtd have access to the ssh-agent" and
also for noticing I didn't refresh the storage pool on the destination
host while doing multiple migration tests. Speak of messy test
environments. . .
Sorry for the noise. At-least I'm now clear on a couple of migration
techniques that libvirt offers. :-)
Have a nice weekend!
--
/kashyap
3551
days inactive
3551
days old
3 comments
1 participants
participants (1)
-
Kashyap Chamarthy