unable to migrate non shared storage in tunneled mode
Hey all, With libvirt 6.5.0 and qemu 5.1.0 migration of non shared disks in tunneled mode does not work for me: virsh # migrate alpinelinux3.8 qemu+tls://ratchet.lan/system --live --persistent --undefinesource --copy-storage-all --tunneled --p2p error: internal error: qemu unexpectedly closed the monitor: Receiving block device images Error unknown block device 2020-08-15T21:21:48.995016Z qemu-system-x86_64: error while loading state section id 1(block) 2020-08-15T21:21:48.995193Z qemu-system-x86_64: load of migration failed: Invalid argument This is both with UEFI and BIOS guests. I understand that newer ways of migrating non shared disks is via NBD directly with QEMU, however I am certain that this used to work before libvirt 6.0. There is a series of commits to /src/qemu/qemu_migration.c on Dec 8, 2019, could they have something to do with this? Is migration of non shared disks supported and supposed to work in tunneled mode or is it not a supported configuration and native NBD directly with QEMU should be used in all cases? Thanks in advance! Full qemu log on receiving host: 2020-08-15 21:23:38.917+0000: starting up libvirt version: 6.5.0, qemu version: 5.1.0, kernel: 5.4.57-gentoo, hostname: ratchet.lan LC_ALL=C \ PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin \ HOME=/var/lib/libvirt/qemu/domain-4-alpinelinux3.8 \ USER=root \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-4-alpinelinux3.8/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-4-alpinelinux3.8/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-4-alpinelinux3.8/.config \ QEMU_AUDIO_DRV=none \ /usr/bin/qemu-system-x86_64 \ -name guest=alpinelinux3.8,debug-threads=on \ -S \ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-alpinelinux3.8/master-key.aes \ -blockdev '{"driver":"file","filename":"/var/lib/libvirt/uefi/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/alpinelinux3.8_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \ -machine pc-q35-5.1,accel=kvm,usb=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \ -cpu kvm64,ibpb=on,md-clear=on,spec-ctrl=on,ssbd=on,vme=on,x2apic=on,hypervisor=on \ -m 2048 \ -overcommit mem-lock=off \ -smp 2,sockets=2,cores=1,threads=1 \ -uuid 95286971-32fa-4138-be0e-519ec21af800 \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=35,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -global ICH9-LPC.disable_s3=1 \ -global ICH9-LPC.disable_s4=1 \ -boot strict=on \ -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.1,addr=0x0 \ -blockdev '{"driver":"host_device","filename":"/dev/vg0/test","aio":"native","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -device virtio-blk-pci,bus=pci.2,addr=0x0,drive=libvirt-2-format,id=virtio-disk0,bootindex=1,write-cache=on \ -device ide-cd,bus=ide.0,id=sata0-0-0 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -vnc 127.0.0.1:2 \ -device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \ -incoming defer \ -device virtio-balloon-pci,id=balloon0,bus=pci.3,addr=0x0 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/5 (label charserial0) Receiving block device images Error unknown block device 2020-08-15T21:23:39.278287Z qemu-system-x86_64: error while loading state section id 1(block) 2020-08-15T21:23:39.278422Z qemu-system-x86_64: load of migration failed: Invalid argument 2020-08-15 21:23:39.344+0000: shutting down, reason=failed
On Sat, Aug 15, 2020 at 15:38:19 -0700, Vjaceslavs Klimovs wrote:
Hey all, With libvirt 6.5.0 and qemu 5.1.0 migration of non shared disks in tunneled mode does not work for me:
virsh # migrate alpinelinux3.8 qemu+tls://ratchet.lan/system --live --persistent --undefinesource --copy-storage-all --tunneled --p2p error: internal error: qemu unexpectedly closed the monitor: Receiving block device images Error unknown block device 2020-08-15T21:21:48.995016Z qemu-system-x86_64: error while loading state section id 1(block) 2020-08-15T21:21:48.995193Z qemu-system-x86_64: load of migration failed: Invalid argument
This is both with UEFI and BIOS guests.
The migration of storage using the qemu migration stream is not supported by qemu when -blockdev is used. Since we can only transport one stream when migrating with '-tunelled', it's no longer possible to combine those two. Unfortunately automagic block migration was deemed a legacy feature.
I understand that newer ways of migrating non shared disks is via NBD directly with QEMU, however I am certain that this used to work before libvirt 6.0. There is a series of commits to /src/qemu/qemu_migration.c on Dec 8, 2019, could they have something to do with this?
I presume those are my commits which fix the NBD migration though. Unfortunately they can't fix the old one.
Is migration of non shared disks supported and supposed to work in tunneled mode or is it not a supported configuration and native NBD directly with QEMU should be used in all cases?
NBD should be used nowadays. The old style storage migration was neglected for a very long time now.
On Mon, Aug 17, 2020 at 3:24 AM Peter Krempa <pkrempa@redhat.com> wrote:
On Sat, Aug 15, 2020 at 15:38:19 -0700, Vjaceslavs Klimovs wrote:
Hey all, With libvirt 6.5.0 and qemu 5.1.0 migration of non shared disks in tunneled mode does not work for me:
virsh # migrate alpinelinux3.8 qemu+tls://ratchet.lan/system --live --persistent --undefinesource --copy-storage-all --tunneled --p2p error: internal error: qemu unexpectedly closed the monitor: Receiving block device images Error unknown block device 2020-08-15T21:21:48.995016Z qemu-system-x86_64: error while loading state section id 1(block) 2020-08-15T21:21:48.995193Z qemu-system-x86_64: load of migration failed: Invalid argument
This is both with UEFI and BIOS guests.
The migration of storage using the qemu migration stream is not supported by qemu when -blockdev is used. Since we can only transport one stream when migrating with '-tunelled', it's no longer possible to combine those two. Unfortunately automagic block migration was deemed a legacy feature.
I understand that newer ways of migrating non shared disks is via NBD directly with QEMU, however I am certain that this used to work before libvirt 6.0. There is a series of commits to /src/qemu/qemu_migration.c on Dec 8, 2019, could they have something to do with this?
I presume those are my commits which fix the NBD migration though. Unfortunately they can't fix the old one.
Is migration of non shared disks supported and supposed to work in tunneled mode or is it not a supported configuration and native NBD directly with QEMU should be used in all cases?
NBD should be used nowadays. The old style storage migration was neglected for a very long time now.
Thank you for the clarification, very helpful. Local libvirtd of course knows about --tunneled and -blockdev being used - would it be possible for it to provide a clear error message that this combination is not supported when such migration is attempted?
I've filed https://gitlab.com/libvirt/libvirt/-/issues/65 for this issue, thanks again for your help. On Mon, Aug 17, 2020 at 7:38 PM Vjaceslavs Klimovs <vklimovs@gmail.com> wrote:
On Mon, Aug 17, 2020 at 3:24 AM Peter Krempa <pkrempa@redhat.com> wrote:
On Sat, Aug 15, 2020 at 15:38:19 -0700, Vjaceslavs Klimovs wrote:
Hey all, With libvirt 6.5.0 and qemu 5.1.0 migration of non shared disks in tunneled mode does not work for me:
virsh # migrate alpinelinux3.8 qemu+tls://ratchet.lan/system --live --persistent --undefinesource --copy-storage-all --tunneled --p2p error: internal error: qemu unexpectedly closed the monitor: Receiving block device images Error unknown block device 2020-08-15T21:21:48.995016Z qemu-system-x86_64: error while loading state section id 1(block) 2020-08-15T21:21:48.995193Z qemu-system-x86_64: load of migration failed: Invalid argument
This is both with UEFI and BIOS guests.
The migration of storage using the qemu migration stream is not supported by qemu when -blockdev is used. Since we can only transport one stream when migrating with '-tunelled', it's no longer possible to combine those two. Unfortunately automagic block migration was deemed a legacy feature.
I understand that newer ways of migrating non shared disks is via NBD directly with QEMU, however I am certain that this used to work before libvirt 6.0. There is a series of commits to /src/qemu/qemu_migration.c on Dec 8, 2019, could they have something to do with this?
I presume those are my commits which fix the NBD migration though. Unfortunately they can't fix the old one.
Is migration of non shared disks supported and supposed to work in tunneled mode or is it not a supported configuration and native NBD directly with QEMU should be used in all cases?
NBD should be used nowadays. The old style storage migration was neglected for a very long time now.
Thank you for the clarification, very helpful.
Local libvirtd of course knows about --tunneled and -blockdev being used - would it be possible for it to provide a clear error message that this combination is not supported when such migration is attempted?
participants (2)
-
Peter Krempa -
Vjaceslavs Klimovs