8:55 a.m.
I've been playing with the new block-migrate feature, but am unable to
get it to work.
[root@src ~]# virsh migrate --live --p2p --tunnelled
--copy-storage-all 4c5c75b9-decc-41c9-9296-20ca5bd5c355
qemu://dst/system
error: Unknown failure
/var/log/libvirt/qemu/4c5c75b9-decc-41c9-9296-20ca5bd5c355.log on the
destination host shows:
bind(unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355):
Permission denied
Migration failed. Exit code
unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355(-22),
exiting
It seems that qemu is not able to write to that location.
[root@dst qemu]# ls -ld /var/run/libvirt/qemu/
drwx------. 2 root root 4096 Aug 7 15:40 /var/run/libvirt/qemu/
As a workaround I gave qemu write permission, and now the block migrate starts.
Receiving block device images
Completed 100 %
Then the migrate command fails:
[root@src ~]# virsh migrate --live --p2p --tunnelled
--copy-storage-all 4c5c75b9-decc-41c9-9296-20ca5bd5c355
qemu://phy004.tilaa.nl/system
error: Unknown failure
and the only thing I can find is in /var/log/libvirt/libvirt.log on
the destination:
15:40:12.370: error : qemuStreamMigWrite:10376 : cannot write to
stream: Broken pipe
Maybe I'm using the wrong options for virsh migrate, but they're not
described in the manpage.
Any hints?
Regards,
Ruben Kerkhof
3:30 p.m.
On 08/07/2010 11:55 PM, Ruben Kerkhof wrote:
<snip>
Maybe I'm using the wrong options for virsh migrate, but
they're not
described in the manpage.
Any hints?
Hi Ruben,
Haven't tried this stuff myself yet, but the error you're mentioning
look similar to what can happen if SELinux is enabled and blocking
things. Same for AppArmor too I guess.
Is there any chance that might be the problem?
Regards and best wishes,
Justin Clift
Regards,
Ruben Kerkhof
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
4:57 p.m.
Hi Justin,
On Mon, Aug 9, 2010 at 22:30, Justin Clift <jclift(a)redhat.com> wrote:
On 08/07/2010 11:55 PM, Ruben Kerkhof wrote:
<snip>
>
> Maybe I'm using the wrong options for virsh migrate, but they're not
> described in the manpage.
>
> Any hints?
Hi Ruben,
Haven't tried this stuff myself yet, but the error you're mentioning look
similar to what can happen if SELinux is enabled and blocking things. Same
for AppArmor too I guess.
Is there any chance that might be the problem?
Selinux is running in permissive mode, and I don't see any related
AVC's in my audit log.
I'm on Fedora 13 by the way, using libvirt 0.8.3. I've tried this with
both qemu-kvm 0.12.3 and a git shapshot of the upcoming qemu-kvm
0.13.
Regards and best wishes,
Justin Clift
Thanks for the help,
Ruben
5:18 a.m.
On Mon, Aug 09, 2010 at 11:57:34PM +0200, Ruben Kerkhof wrote:
Hi Justin,
On Mon, Aug 9, 2010 at 22:30, Justin Clift <jclift(a)redhat.com> wrote:
> On 08/07/2010 11:55 PM, Ruben Kerkhof wrote:
> <snip>
>>
>> Maybe I'm using the wrong options for virsh migrate, but they're not
>> described in the manpage.
>>
>> Any hints?
>
> Hi Ruben,
>
> Haven't tried this stuff myself yet, but the error you're mentioning look
> similar to what can happen if SELinux is enabled and blocking things. Same
> for AppArmor too I guess.
>
> Is there any chance that might be the problem?
Selinux is running in permissive mode, and I don't see any related
AVC's in my audit log.
I'm on Fedora 13 by the way, using libvirt 0.8.3. I've tried this with
both qemu-kvm 0.12.3 and a git shapshot of the upcoming qemu-kvm
0.13.
Is the error lso showing up if you're using a shared storage for
migration ? A couple of things to look for are DNS entries, make sure
both source and target are in the DNS and resolve to their IP addresses,
also check iptables !
Daniel
--
Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/
daniel(a)veillard.com | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library http://libvirt.org/
7:16 a.m.
On Sat, Aug 07, 2010 at 03:55:10PM +0200, Ruben Kerkhof wrote:
I've been playing with the new block-migrate feature, but am
unable to
get it to work.
[root@src ~]# virsh migrate --live --p2p --tunnelled
--copy-storage-all 4c5c75b9-decc-41c9-9296-20ca5bd5c355
qemu://dst/system
error: Unknown failure
/var/log/libvirt/qemu/4c5c75b9-decc-41c9-9296-20ca5bd5c355.log on the
destination host shows:
bind(unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355):
Permission denied
Migration failed. Exit code
unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355(-22),
exiting
It seems that qemu is not able to write to that location.
[root@dst qemu]# ls -ld /var/run/libvirt/qemu/
drwx------. 2 root root 4096 Aug 7 15:40 /var/run/libvirt/qemu/
As a workaround I gave qemu write permission, and now the block migrate starts.
Did the disk image files exist on the target before migration started ?
If not, then this is going to be falling foul of our security drivers
which prevent QEMU creating new files itself.
Also, a straw poll of people at the KVM forum yesterday determined that no
one had ever got block migration to work, so it might be just broken in
QEMU
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
7:22 a.m.
Hi Daniel,
On Wed, Aug 11, 2010 at 14:16, Daniel P. Berrange <berrange(a)redhat.com> wrote:
On Sat, Aug 07, 2010 at 03:55:10PM +0200, Ruben Kerkhof wrote:
> I've been playing with the new block-migrate feature, but am unable to
> get it to work.
>
> [root@src ~]# virsh migrate --live --p2p --tunnelled
> --copy-storage-all 4c5c75b9-decc-41c9-9296-20ca5bd5c355
> qemu://dst/system
> error: Unknown failure
>
> /var/log/libvirt/qemu/4c5c75b9-decc-41c9-9296-20ca5bd5c355.log on the
> destination host shows:
>
>
bind(unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355):
> Permission denied
> Migration failed. Exit code
>
unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355(-22),
> exiting
>
> It seems that qemu is not able to write to that location.
>
> [root@dst qemu]# ls -ld /var/run/libvirt/qemu/
> drwx------. 2 root root 4096 Aug 7 15:40 /var/run/libvirt/qemu/
>
> As a workaround I gave qemu write permission, and now the block migrate starts.
Did the disk image files exist on the target before migration started ?
If not, then this is going to be falling foul of our security drivers
which prevent QEMU creating new files itself.
I'm working with raw lvs and created them on the destination before
starting the migration.
Also, a straw poll of people at the KVM forum yesterday determined
that no
one had ever got block migration to work, so it might be just broken in
QEMU
Daniel
Actually, I've got it working now, with the --direct option and the
--p2p --direct option.
The tunneled method fails though. I'm trying to debug this now.
The first thing which strikes me as odd is in the
qemuMonitorTextMigrate function:
1173 if (qemuMonitorCommand(mon, cmd, &info) < 0) {
(gdb) p cmd
$4 = 0x7fffdc0e6ef0 "migrate -d
-b\"unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.src.4c5c75b9-decc-41c9-9296-20ca5bd5c355\""
(gdb) n
1180 if (strstr(info, "fail") != NULL) {
(gdb) n
1181 qemuReportError(VIR_ERR_OPERATION_FAILED,
(gdb) p info
$5 = 0x7fffe8001210 "migration failed\r\n"
Not sure if there's supposed to be a space between the migrate and the
-d option, and no space after the -b.
Regards,
Ruben
8:35 a.m.
On Wed, Aug 11, 2010 at 14:22, Ruben Kerkhof <ruben(a)rubenkerkhof.com> wrote:
Actually, I've got it working now, with the --direct option and
the
--p2p --direct option.
The tunneled method fails though. I'm trying to debug this now.
I've got the tunnelled method working now as well, but had to disable
the security driver on the destination host, and run qemu as
root/root.
Regards,
Ruben
11:08 a.m.
On 08/11/2010 11:35 PM, Ruben Kerkhof wrote:
On Wed, Aug 11, 2010 at 14:22, Ruben
Kerkhof<ruben(a)rubenkerkhof.com> wrote:
> Actually, I've got it working now, with the --direct option and the
> --p2p --direct option.
> The tunneled method fails though. I'm trying to debug this now.
I've got the tunnelled method working now as well, but had to disable
the security driver on the destination host, and run qemu as
root/root.
Hi Ruben,
Would you have time to add the steps / command line arguments you used
to a page on the wiki?
Writing it up in a fuller fashion would take some time, but it sounds
like you have the steps for making it work understood, so even basic
info to help people would be very cool. :)
?
Regards and best wishes,
Justin Clift
11:18 a.m.
On Wed, Aug 11, 2010 at 18:08, Justin Clift <jclift(a)redhat.com> wrote:
Would you have time to add the steps / command line arguments you
used to a
page on the wiki?
Writing it up in a fuller fashion would take some time, but it sounds like
you have the steps for making it work understood, so even basic info to help
people would be very cool. :)
Sure, I'll write something up, and see if I can come up with a patch
for the virsh manpage as well.
Regards,
Ruben
1:53 a.m.
On 08/11/2010 11:35 PM, Ruben Kerkhof wrote:
On Wed, Aug 11, 2010 at 14:22, Ruben
Kerkhof<ruben(a)rubenkerkhof.com> wrote:
> Actually, I've got it working now, with the --direct option and the
> --p2p --direct option.
> The tunneled method fails though. I'm trying to debug this now.
I've got the tunnelled method working now as well, but had to disable
the security driver on the destination host, and run qemu as
root/root.
Hi Ruben,
Chris Lalancette was kind enough to look into why libvirt was requiring
root:root QEMU ownership for tunnelled migration to work. He found the
problem (temporary socket file being located in root owned directory)
and submitted the patch last night.
The next release of libvirt (0.8.4) should therefore allow qemu:qemu
owned QEMU to work. :)
Note, I only tested the standard "shared storage" tunnelled migration,
rather than block migration too. Personally not sure if there are
further gotchas in block migration code waiting to be found. ;)
Hope that helps.
Regards and best wishes,
Justin Clift
12:28 p.m.
Hi Justin,
On Fri, Aug 13, 2010 at 08:53, Justin Clift <jclift(a)redhat.com> wrote:
Hi Ruben,
Chris Lalancette was kind enough to look into why libvirt was requiring
root:root QEMU ownership for tunnelled migration to work. He found the
problem (temporary socket file being located in root owned directory) and
submitted the patch last night.
Perfect, thanks a lot.
The next release of libvirt (0.8.4) should therefore allow qemu:qemu owned
QEMU to work. :)
Note, I only tested the standard "shared storage" tunnelled migration,
rather than block migration too.
I've just tested those patches with block-migrate (only the
--copy-storage-all option) and they work fine.
Personally not sure if there are further
gotchas in block migration code waiting to be found. ;)
One issue is that during the block-migrate, virsh list simply hangs,
and some python scripts that I'm using which talk to the libvirt api
return with "Timed out during operation: cannot acquire state change
lock".
Haven't looked at the code yet, but I suspect that the block migration
is holding a lock longer than a normal migration on shared storage
would.
The other issue has nothing to do with block migrate, but migration in
general. It's sometimes hard to see if the error message virsh migrate
is returning is generated on the source or the destination side.
Hope that helps.
Regards and best wishes,
Justin Clift
Thanks again,
Ruben
5215
days inactive
5221
days old
10 comments
4 participants
participants (4)
-
Daniel P. Berrange -
Daniel Veillard -
Justin Clift -
Ruben Kerkhof