[libvirt-users] P2P live migration with non-shared storage: fails to connect to remote libvirt URI qemu+ssh
Migration without --p2p works just fine, ie. the below works: $ virsh migrate --verbose --copy-storage-all \ --live cvm1 qemu+ssh://kashyapc@devstack3/system Migration: [100 %] Result: - On the source host, the guest is shut off - On the destination host, the guest is live migratied successfully Migration with "--p2p" fails, a simple test below: First, I should note, I didn't modify any settings in /etc/libvirt/libvirtd.conf on both source and destination hosts, except for libvirt logging filters. (0) On source and destination hosts, SSH keys are setup so that passwordless auth works: $ ssh-keygen -t $ eval `ssh-agent` $ ssh-add .ssh/id_rsa $ ssh-copy-id root@devstack3 (1) Check if the connection to the remote host works w/o a prompt for user credentials (the below works as user and root): $ virsh -c qemu+ssh://kashyapc@devstack3/system Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # (2) Perform peer to peer live migration (as root): $ virsh migrate --verbose --p2p --copy-storage-all \ --live cvm1 qemu+ssh://kashyapc@devstack3/system error: operation failed: Failed to connect to remote libvirt URI qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection reset by peer
From libvirtd debug log:
[. . .] 2015-04-03 06:04:16.221+0000: 31009: debug : virCommandRunAsync:2408 : About to run LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin ssh -l kashyapc devstack3 sh -c ''\''if '\ ''nc'\'' -q 2>&1 | grep "requires an argument" >/dev/null 2>&1; then ARG=-q0;else ARG=;fi;'\''nc'\'' $ARG -U /var/run/libvirt/libvirt-sock'\''' 2015-04-03 06:04:16.223+0000: 31009: debug : virCommandRunAsync:2411 : Command result 0, with PID 11204 2015-04-03 06:04:16.300+0000: 31009: error : virNetSocketReadWire:1564 : Cannot recv data: Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection reset by peer 2015-04-03 06:04:16.300+0000: 31009: debug : do_open:1194 : driver 6 remote returned ERROR 2015-04-03 06:04:16.300+0000: 31009: debug : qemuDomainObjExitRemote:1695 : Exited remote (vm=0x7f727c005f80 name=cvm1) 2015-04-03 06:04:16.300+0000: 31009: error : doPeer2PeerMigrate:4711 : operation failed: Failed to connect to remote libvirt URI qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection reset by peer 2015-04-03 06:04:16.300+0000: 31009: debug : qemuMigrationRestoreDomainState:1429 : driver=0x7f728c160980, vm=0x7f727c005f80, pre-mig-state=1, state=1 2015-04-03 06:04:16.300+0000: 31009: debug : qemuDomainObjEndAsyncJob:1497 : Stopping async job: migration out (vm=0x7f727c005f80 name=cvm1) 2015-04-03 06:04:16.301+0000: 31007: debug : virProcessAbort:167 : aborting child process 11204 2015-04-03 06:04:16.301+0000: 31007: debug : virProcessAbort:175 : trying SIGTERM to child process 11204 [. . .] What else am I missing? -- /kashyap
On Fri, Apr 03, 2015 at 10:08:21AM +0200, Kashyap Chamarthy wrote:
Migration without --p2p works just fine, ie. the below works:
$ virsh migrate --verbose --copy-storage-all \ --live cvm1 qemu+ssh://kashyapc@devstack3/system Migration: [100 %]
[. . .]
(2) Perform peer to peer live migration (as root):
$ virsh migrate --verbose --p2p --copy-storage-all \ --live cvm1 qemu+ssh://kashyapc@devstack3/system
error: operation failed: Failed to connect to remote libvirt URI qemu+ssh://kashyapc@devstack3/system: Cannot recv data: Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).: Connection reset by peer
Just to note, both the tests were performed with the below versions on source and destination hosts: kernel-4.0.0-0.rc5.git4.1.fc22.x86_64 libvirt-daemon-kvm-1.2.13-2.fc22.x86_64 qemu-system-x86-2.3.0-0.2.rc1.fc22.x86_64 [. . .] -- /kashyap
On Fri, Apr 03, 2015 at 10:13:38AM +0200, Kashyap Chamarthy wrote:
On Fri, Apr 03, 2015 at 10:08:21AM +0200, Kashyap Chamarthy wrote:
Migration without --p2p works just fine, ie. the below works:
$ virsh migrate --verbose --copy-storage-all \ --live cvm1 qemu+ssh://kashyapc@devstack3/system Migration: [100 %]
[. . .]
(2) Perform peer to peer live migration (as root):
$ virsh migrate --verbose --p2p --copy-storage-all \ --live cvm1 qemu+ssh://kashyapc@devstack3/system
Hmm, reading the flow for peer2peer here: https://libvirt.org/migration.html#flowpeer2peer which says: "Note that the source libvirtd uses its own credentials (typically root) to connect to the destination, rather than the credentials used by the client to connect to the source; if these differ, it is common to run into a situation where a client can connect to the destination directly but the source cannot make the connection to set up the peer-to-peer migration." So, somehow libvirtd daemon on source is not able to pick the credentials. I'll try qemu+tcp with the below configs in libvirtd.conf (note: the auth_tcp = "none" because, I'm doing the test in a trusted network). listen_tls = 0 listen_tcp = 1 auth_tcp = "none" -- /kashyap
On Fri, Apr 03, 2015 at 12:27:07PM +0200, Kashyap Chamarthy wrote:
On Fri, Apr 03, 2015 at 10:13:38AM +0200, Kashyap Chamarthy wrote:
On Fri, Apr 03, 2015 at 10:08:21AM +0200, Kashyap Chamarthy wrote:
Migration without --p2p works just fine, ie. the below works:
$ virsh migrate --verbose --copy-storage-all \ --live cvm1 qemu+ssh://kashyapc@devstack3/system Migration: [100 %]
Problem found: I was running the `ssh-agent` with a passphrase for the SSH key. This will work in the managed direct migration case as above (because the client handles process controls most of it). . .
[. . .]
(2) Perform peer to peer live migration (as root):
$ virsh migrate --verbose --p2p --copy-storage-all \ --live cvm1 qemu+ssh://kashyapc@devstack3/system
. . .but _not_ in P2P case, because the libvirt daemon on the source _does not_ have access to the `ssh-agent` (so while I _can_ SSH into the dest host as 'root' w/o password, libvirt daemon cannot! So ensure that SSH key has no pass-phrase (only in trusted networks for testing only!) or setup X.509 certificates. When Jirka asked me on IRC that if I was able to SSH as root to the destination host, I was able to, but didn't think of the `ssh-agent` holding the private key _with_ passphrase!
Hmm, reading the flow for peer2peer here:
https://libvirt.org/migration.html#flowpeer2peer
which says:
"Note that the source libvirtd uses its own credentials (typically root) to connect to the destination, rather than the credentials used by the client to connect to the source; if these differ, it is common to run into a situation where a client can connect to the destination directly but the source cannot make the connection to set up the peer-to-peer migration."
Also thanks to Lars Kellogg-Stedman (OpenStack dev) who reminded me of the the question "does your libvirtd have access to the ssh-agent" and also for noticing I didn't refresh the storage pool on the destination host while doing multiple migration tests. Speak of messy test environments. . . Sorry for the noise. At-least I'm now clear on a couple of migration techniques that libvirt offers. :-) Have a nice weekend! -- /kashyap
participants (1)
-
Kashyap Chamarthy