I've been playing with the new block-migrate feature, but am unable to get it to work. [root@src ~]# virsh migrate --live --p2p --tunnelled --copy-storage-all 4c5c75b9-decc-41c9-9296-20ca5bd5c355 qemu://dst/system error: Unknown failure /var/log/libvirt/qemu/4c5c75b9-decc-41c9-9296-20ca5bd5c355.log on the destination host shows: bind(unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355): Permission denied Migration failed. Exit code unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355(-22), exiting It seems that qemu is not able to write to that location. [root@dst qemu]# ls -ld /var/run/libvirt/qemu/ drwx------. 2 root root 4096 Aug 7 15:40 /var/run/libvirt/qemu/ As a workaround I gave qemu write permission, and now the block migrate starts. Receiving block device images Completed 100 % Then the migrate command fails: [root@src ~]# virsh migrate --live --p2p --tunnelled --copy-storage-all 4c5c75b9-decc-41c9-9296-20ca5bd5c355 qemu://phy004.tilaa.nl/system error: Unknown failure and the only thing I can find is in /var/log/libvirt/libvirt.log on the destination: 15:40:12.370: error : qemuStreamMigWrite:10376 : cannot write to stream: Broken pipe Maybe I'm using the wrong options for virsh migrate, but they're not described in the manpage. Any hints? Regards, Ruben Kerkhof
On 08/07/2010 11:55 PM, Ruben Kerkhof wrote: <snip>
Maybe I'm using the wrong options for virsh migrate, but they're not described in the manpage.
Any hints?
Hi Ruben, Haven't tried this stuff myself yet, but the error you're mentioning look similar to what can happen if SELinux is enabled and blocking things. Same for AppArmor too I guess. Is there any chance that might be the problem? Regards and best wishes, Justin Clift
Regards,
Ruben Kerkhof
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Hi Justin, On Mon, Aug 9, 2010 at 22:30, Justin Clift <jclift@redhat.com> wrote:
On 08/07/2010 11:55 PM, Ruben Kerkhof wrote: <snip>
Maybe I'm using the wrong options for virsh migrate, but they're not described in the manpage.
Any hints?
Hi Ruben,
Haven't tried this stuff myself yet, but the error you're mentioning look similar to what can happen if SELinux is enabled and blocking things. Same for AppArmor too I guess.
Is there any chance that might be the problem?
Selinux is running in permissive mode, and I don't see any related AVC's in my audit log. I'm on Fedora 13 by the way, using libvirt 0.8.3. I've tried this with both qemu-kvm 0.12.3 and a git shapshot of the upcoming qemu-kvm 0.13.
Regards and best wishes,
Justin Clift
Thanks for the help, Ruben
On Mon, Aug 09, 2010 at 11:57:34PM +0200, Ruben Kerkhof wrote:
Hi Justin,
On Mon, Aug 9, 2010 at 22:30, Justin Clift <jclift@redhat.com> wrote:
On 08/07/2010 11:55 PM, Ruben Kerkhof wrote: <snip>
Maybe I'm using the wrong options for virsh migrate, but they're not described in the manpage.
Any hints?
Hi Ruben,
Haven't tried this stuff myself yet, but the error you're mentioning look similar to what can happen if SELinux is enabled and blocking things. Same for AppArmor too I guess.
Is there any chance that might be the problem?
Selinux is running in permissive mode, and I don't see any related AVC's in my audit log. I'm on Fedora 13 by the way, using libvirt 0.8.3. I've tried this with both qemu-kvm 0.12.3 and a git shapshot of the upcoming qemu-kvm 0.13.
Is the error lso showing up if you're using a shared storage for migration ? A couple of things to look for are DNS entries, make sure both source and target are in the DNS and resolve to their IP addresses, also check iptables ! Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
On Sat, Aug 07, 2010 at 03:55:10PM +0200, Ruben Kerkhof wrote:
I've been playing with the new block-migrate feature, but am unable to get it to work.
[root@src ~]# virsh migrate --live --p2p --tunnelled --copy-storage-all 4c5c75b9-decc-41c9-9296-20ca5bd5c355 qemu://dst/system error: Unknown failure
/var/log/libvirt/qemu/4c5c75b9-decc-41c9-9296-20ca5bd5c355.log on the destination host shows:
bind(unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355): Permission denied Migration failed. Exit code unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355(-22), exiting
It seems that qemu is not able to write to that location.
[root@dst qemu]# ls -ld /var/run/libvirt/qemu/ drwx------. 2 root root 4096 Aug 7 15:40 /var/run/libvirt/qemu/
As a workaround I gave qemu write permission, and now the block migrate starts.
Did the disk image files exist on the target before migration started ? If not, then this is going to be falling foul of our security drivers which prevent QEMU creating new files itself. Also, a straw poll of people at the KVM forum yesterday determined that no one had ever got block migration to work, so it might be just broken in QEMU Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Hi Daniel, On Wed, Aug 11, 2010 at 14:16, Daniel P. Berrange <berrange@redhat.com> wrote:
On Sat, Aug 07, 2010 at 03:55:10PM +0200, Ruben Kerkhof wrote:
I've been playing with the new block-migrate feature, but am unable to get it to work.
[root@src ~]# virsh migrate --live --p2p --tunnelled --copy-storage-all 4c5c75b9-decc-41c9-9296-20ca5bd5c355 qemu://dst/system error: Unknown failure
/var/log/libvirt/qemu/4c5c75b9-decc-41c9-9296-20ca5bd5c355.log on the destination host shows:
bind(unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355): Permission denied Migration failed. Exit code unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.4c5c75b9-decc-41c9-9296-20ca5bd5c355(-22), exiting
It seems that qemu is not able to write to that location.
[root@dst qemu]# ls -ld /var/run/libvirt/qemu/ drwx------. 2 root root 4096 Aug 7 15:40 /var/run/libvirt/qemu/
As a workaround I gave qemu write permission, and now the block migrate starts.
Did the disk image files exist on the target before migration started ? If not, then this is going to be falling foul of our security drivers which prevent QEMU creating new files itself.
I'm working with raw lvs and created them on the destination before starting the migration.
Also, a straw poll of people at the KVM forum yesterday determined that no one had ever got block migration to work, so it might be just broken in QEMU
Daniel
Actually, I've got it working now, with the --direct option and the --p2p --direct option. The tunneled method fails though. I'm trying to debug this now. The first thing which strikes me as odd is in the qemuMonitorTextMigrate function: 1173 if (qemuMonitorCommand(mon, cmd, &info) < 0) { (gdb) p cmd $4 = 0x7fffdc0e6ef0 "migrate -d -b\"unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.src.4c5c75b9-decc-41c9-9296-20ca5bd5c355\"" (gdb) n 1180 if (strstr(info, "fail") != NULL) { (gdb) n 1181 qemuReportError(VIR_ERR_OPERATION_FAILED, (gdb) p info $5 = 0x7fffe8001210 "migration failed\r\n" Not sure if there's supposed to be a space between the migrate and the -d option, and no space after the -b. Regards, Ruben
On Wed, Aug 11, 2010 at 14:22, Ruben Kerkhof <ruben@rubenkerkhof.com> wrote:
Actually, I've got it working now, with the --direct option and the --p2p --direct option. The tunneled method fails though. I'm trying to debug this now.
I've got the tunnelled method working now as well, but had to disable the security driver on the destination host, and run qemu as root/root. Regards, Ruben
On 08/11/2010 11:35 PM, Ruben Kerkhof wrote:
On Wed, Aug 11, 2010 at 14:22, Ruben Kerkhof<ruben@rubenkerkhof.com> wrote:
Actually, I've got it working now, with the --direct option and the --p2p --direct option. The tunneled method fails though. I'm trying to debug this now.
I've got the tunnelled method working now as well, but had to disable the security driver on the destination host, and run qemu as root/root.
Hi Ruben, Would you have time to add the steps / command line arguments you used to a page on the wiki? Writing it up in a fuller fashion would take some time, but it sounds like you have the steps for making it work understood, so even basic info to help people would be very cool. :) ? Regards and best wishes, Justin Clift
On Wed, Aug 11, 2010 at 18:08, Justin Clift <jclift@redhat.com> wrote:
Would you have time to add the steps / command line arguments you used to a page on the wiki?
Writing it up in a fuller fashion would take some time, but it sounds like you have the steps for making it work understood, so even basic info to help people would be very cool. :)
Sure, I'll write something up, and see if I can come up with a patch for the virsh manpage as well. Regards, Ruben
On 08/11/2010 11:35 PM, Ruben Kerkhof wrote:
On Wed, Aug 11, 2010 at 14:22, Ruben Kerkhof<ruben@rubenkerkhof.com> wrote:
Actually, I've got it working now, with the --direct option and the --p2p --direct option. The tunneled method fails though. I'm trying to debug this now.
I've got the tunnelled method working now as well, but had to disable the security driver on the destination host, and run qemu as root/root.
Hi Ruben, Chris Lalancette was kind enough to look into why libvirt was requiring root:root QEMU ownership for tunnelled migration to work. He found the problem (temporary socket file being located in root owned directory) and submitted the patch last night. The next release of libvirt (0.8.4) should therefore allow qemu:qemu owned QEMU to work. :) Note, I only tested the standard "shared storage" tunnelled migration, rather than block migration too. Personally not sure if there are further gotchas in block migration code waiting to be found. ;) Hope that helps. Regards and best wishes, Justin Clift
Hi Justin, On Fri, Aug 13, 2010 at 08:53, Justin Clift <jclift@redhat.com> wrote:
Hi Ruben,
Chris Lalancette was kind enough to look into why libvirt was requiring root:root QEMU ownership for tunnelled migration to work. He found the problem (temporary socket file being located in root owned directory) and submitted the patch last night.
Perfect, thanks a lot.
The next release of libvirt (0.8.4) should therefore allow qemu:qemu owned QEMU to work. :)
Note, I only tested the standard "shared storage" tunnelled migration, rather than block migration too.
I've just tested those patches with block-migrate (only the --copy-storage-all option) and they work fine.
Personally not sure if there are further gotchas in block migration code waiting to be found. ;)
One issue is that during the block-migrate, virsh list simply hangs, and some python scripts that I'm using which talk to the libvirt api return with "Timed out during operation: cannot acquire state change lock". Haven't looked at the code yet, but I suspect that the block migration is holding a lock longer than a normal migration on shared storage would. The other issue has nothing to do with block migrate, but migration in general. It's sometimes hard to see if the error message virsh migrate is returning is generated on the source or the destination side.
Hope that helps.
Regards and best wishes,
Justin Clift
Thanks again, Ruben
participants (4)
-
Daniel P. Berrange -
Daniel Veillard -
Justin Clift -
Ruben Kerkhof