[libvirt-users] virsh migration job unexpectedly failed

Hi , when i have run virsh migrate command , the migration progress fiailed at [61 %]. command : virsh migrate --live --undefinesource --p2p --copy-storage-inc --verbose --persistent instance-00001959 qemu+tcp://host292/system result : Migration: [ 61 %]error: operation failed: migration job: unexpectedly failed os/kernel version: centos 6.7 / kernel 2.6.32 qemu-kvm version : 0.12.1.2 libvirt version : 0.10.2-54 the libvirt.log on target host: 2018-05-22 11:45:38.603+0000: 32314: warning : qemuDomainObjTaint:1459 : Domain id=30 name='instance-00001959' uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 is tainted: high-privileges 2018-05-22 11:45:38.883+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:38.902+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:38.903+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.090+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.090+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.091+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:41.841+0000: 32310: error : qemuMonitorIO:615 : internal error End of file from monitor the qmeu-kvminstance-00001959.log: 2018-05-22 11:45:38.603+0000: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/home/ning.zhao USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-00001959 -S -M rhel6.6.0 -cpu Broadwell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+m$ nitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 41975d30-2490-473e-be7b-9d2d08bf66d2 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.1.2-4.el6,serial=fcbe6c27-c9$ d-e811-ac36-a281f9ebc503,uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00001959.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -n$ -kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/41975d30-2490-473e-be7b-9d2d08bf66d2/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4$ drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:c0:8f:6b,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/41975d30-2490-473e-b$ 7b-9d2d08bf66d2/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -k en-us -vga cirrus -incoming tcp: 0.0.0.0:49152 -device virtio-bal$ oon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on Domain id=30 is tainted: high-privileges char device redirected to /dev/pts/2 Receiving block device images qemu: warning: error while loading state for instance 0x0 of device 'block' load of migration failed 2018-05-22 11:45:41.841+0000: shutting down i checked the log and disk file and they have right permissions, has anyone know about this ,i have google many times ,but i cant find the right debug way to resolve it. thanks

On 05/23/2018 01:26 PM, cokabug wrote:
Hi ,
when i have run virsh migrate command , the migration progress fiailed at [61 %]. command : virsh migrate --live --undefinesource --p2p --copy-storage-inc --verbose --persistent instance-00001959 qemu+tcp://host292/system
result : Migration: [ 61 %]error: operation failed: migration job: unexpectedly failed
os/kernel version: centos 6.7 / kernel 2.6.32 qemu-kvm version : 0.12.1.2 libvirt version : 0.10.2-54
These are very ancient versions. I wouldn't be surprised if you upgrade and find out the problem is fixed.
the libvirt.log on target host: 2018-05-22 11:45:38.603+0000: 32314: warning : qemuDomainObjTaint:1459 : Domain id=30 name='instance-00001959' uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 is tainted: high-privileges 2018-05-22 11:45:38.883+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:38.902+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:38.903+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.090+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.090+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.091+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:41.841+0000: 32310: error : qemuMonitorIO:615 : internal error End of file from monitor
This says that qemu died. You can try to look into domain log to see why. Michal

finally, i found the problem, qemu-img create command not specific vm disk size on target host, when i use the size same as source vm, the migration finish successfully. On Wed, May 23, 2018 at 7:56 PM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 05/23/2018 01:26 PM, cokabug wrote:
Hi ,
when i have run virsh migrate command , the migration progress fiailed at [61 %]. command : virsh migrate --live --undefinesource --p2p --copy-storage-inc --verbose --persistent instance-00001959 qemu+tcp://host292/system
result : Migration: [ 61 %]error: operation failed: migration job: unexpectedly failed
os/kernel version: centos 6.7 / kernel 2.6.32 qemu-kvm version : 0.12.1.2 libvirt version : 0.10.2-54
These are very ancient versions. I wouldn't be surprised if you upgrade and find out the problem is fixed.
the libvirt.log on target host: 2018-05-22 11:45:38.603+0000: 32314: warning :
qemuDomainObjTaint:1459
: Domain id=30 name='instance-00001959' uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 is tainted: high-privileges 2018-05-22 11:45:38.883+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:38.902+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:38.903+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.090+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.090+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.091+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:41.841+0000: 32310: error : qemuMonitorIO:615 : internal error End of file from monitor
This says that qemu died. You can try to look into domain log to see why.
Michal
participants (2)
-
cokabug
-
Michal Privoznik