[libvirt-users] Live migrate failure
Hi everyone, I use KVM on Openstack Cloud. And KVM migrated successfull, but not ping. *Source Host SIDE: *"/var/log/libvirt/libvirtd.log" 2014-06-16 05:15:50.599+0000: 2096: warning : qemuMigrationCancelDriveMirror:1383 : Unable to stop block job on drive-virtio-disk0 2014-06-16 05:15:50.601+0000: 2096: *warning : qemuMigrationCancelDriveMirror:1383 : Unable to stop block job on drive-virtio-disk1* *2014-06-16 05:15:50.651+0000: 2096: error : virCommandWait:2348 : internal error: Child process (ovs-vsctl --timeout=5 get Interface tap6977e4a1-cd external_ids:PortData) unexpected exit status 1: ovs-vsctl: no key "PortData" in Interface record "tap6977e4a1-cd" column external_ids* 2014-06-16 05:15:50.651+0000: 2096: error : virNetDevOpenvswitchGetMigrateData:219 : Unable to run command to get OVS port data for interface tap6977e4a1-cd: Operation not permitted 2014-06-16 05:15:50.651+0000: 2096: error : qemuMigrationCookieNetworkAlloc:367 : Unable to run command to get OVS port data for interface tap6977e4a1-cd: Operation not permitted 2014-06-16 05:15:50.651+0000: 2096: warning : qemuMigrationRun:3262 : Unable to encode migration cookie 2014-06-16 05:15:53.321+0000: 2096: warning : virAuditSend:135 : Failed to send audit message virt=kvm op=stop reason=migrated vm="instance-0001b37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd vm-pid=-1: Operation not permitted *Destination Host SIDE:* "/var/log/libvirt/libvirtd.log" ...... 2014-06-16 05:15:41.419+0000: 2229: warning : virAuditSend:135 : Failed to send audit message virt=kvm resrc=cgroup reason=allow vm="instance-000 1b37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd cgroup="/sys/fs/cgroup/devices/machine/instance-0001b37c.libvirt-qemu/" class=major category=pty maj=88 acl=rw: Operation not permitted 2014-06-16 05:15:41.620+0000: 2229: warning : qemuDomainObjEnterMonitorInternal:1175 : This thread seems to be the async job owner; entering moni tor without asking for a nested job is dangerous 2014-06-16 05:15:43.552+0000: 2229: warning : virAuditSend:135 : Failed to send audit message virt=kvm resrc=vcpu reason=start vm="instance-0001b 37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd old-vcpu=0 new-vcpu=2: Operation not permitted 2014-06-16 05:15:43.552+0000: 2229: warning : virAuditSend:135 :* Failed to send audit message *virt=kvm op=start reason=migrated vm="instance-0001 b37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd vm-pid=21063:* Operation not permitted* 1) I'm using libvirt version 1.1.1, libvirt-bin 1.1.1-0ubuntu8.5~cloud0 libvirt0 1.1.1-0ubuntu8.5~cloud0 python-libvirt 1.1.1-0ubuntu8.5~cloud0 2) #cat /etc/libvirt/qemu.conf ... user = "root" group = "root" I searched GOOGLE, René speaks "downgrading to libvirt 1.1.0 and reinstalling everything. They will ok". But I don't think that. How to resolve this error?. Thanks all! -- TuanTaBa
On 06/16/2014 02:06 AM, Tuân Tạ Bá wrote:
Hi everyone,
I use KVM on Openstack Cloud. And KVM migrated successfull, but not ping.
*Source Host SIDE: *"/var/log/libvirt/libvirtd.log"
2014-06-16 05:15:50.599+0000: 2096: warning : qemuMigrationCancelDriveMirror:1383 : Unable to stop block job on drive-virtio-disk0 2014-06-16 05:15:50.601+0000: 2096: *warning : qemuMigrationCancelDriveMirror:1383 : Unable to stop block job on drive-virtio-disk1*
These warnings don't look good; have you investigated why they are occurring?
*2014-06-16 05:15:50.651+0000: 2096: error : virCommandWait:2348 : internal error: Child process (ovs-vsctl --timeout=5 get Interface tap6977e4a1-cd external_ids:PortData) unexpected exit status 1: ovs-vsctl: no key "PortData" in Interface record "tap6977e4a1-cd" column external_ids*
2014-06-16 05:15:50.651+0000: 2096: error : virNetDevOpenvswitchGetMigrateData:219 : Unable to run command to get OVS port data for interface tap6977e4a1-cd: Operation not permitted 2014-06-16 05:15:50.651+0000: 2096: error : qemuMigrationCookieNetworkAlloc:367 : Unable to run command to get OVS port data for interface tap6977e4a1-cd: Operation not permitted
Based on this failure, migration should not be successful.
2014-06-16 05:15:50.651+0000: 2096: warning : qemuMigrationRun:3262 : Unable to encode migration cookie 2014-06-16 05:15:53.321+0000: 2096: warning : virAuditSend:135 : Failed to send audit message virt=kvm op=stop reason=migrated vm="instance-0001b37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd vm-pid=-1: Operation not permitted
That doesn't look good either. Are you sure your system is set up correctly to allow audit messages to be generated? But it's probably unrelated to the failure you are seeing.
*Destination Host SIDE:* "/var/log/libvirt/libvirtd.log" ......
2014-06-16 05:15:41.419+0000: 2229: warning : virAuditSend:135 : Failed to send audit message virt=kvm resrc=cgroup reason=allow vm="instance-000 1b37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd cgroup="/sys/fs/cgroup/devices/machine/instance-0001b37c.libvirt-qemu/" class=major category=pty maj=88 acl=rw: Operation not permitted 2014-06-16 05:15:41.620+0000: 2229: warning : qemuDomainObjEnterMonitorInternal:1175 : This thread seems to be the async job owner; entering moni tor without asking for a nested job is dangerous 2014-06-16 05:15:43.552+0000: 2229: warning : virAuditSend:135 : Failed to send audit message virt=kvm resrc=vcpu reason=start vm="instance-0001b 37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd old-vcpu=0 new-vcpu=2: Operation not permitted 2014-06-16 05:15:43.552+0000: 2229: warning : virAuditSend:135 :* Failed to send audit message *virt=kvm op=start reason=migrated vm="instance-0001 b37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd vm-pid=21063:* Operation not permitted*
1) I'm using libvirt version 1.1.1,
libvirt-bin 1.1.1-0ubuntu8.5~cloud0 libvirt0 1.1.1-0ubuntu8.5~cloud0 python-libvirt 1.1.1-0ubuntu8.5~cloud0
2) #cat /etc/libvirt/qemu.conf ... user = "root" group = "root"
I searched GOOGLE, René speaks "downgrading to libvirt 1.1.0 and reinstalling everything. They will ok". But I don't think that.
How to resolve this error?.
We've had some bug fixes for migration in the meantime; can you reproduce the error with libvirt 1.2.5? -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Thanks Eric. Sometime I live-migrated sucessfull. Although the following errors still appears. I will retry. *warning : qemuMigrationCancelDriveMirror:1383 : Unable to stop block job on drive-virtio-disk1* *error : virCommandWait:2348 : internal error: Child process (ovs-vsctl --timeout=5 get Interface tap6977e4a1-cd external_ids:PortData) unexpected exit status 1: ovs-vsctl: no key "PortData" in Interface record "tap6977e4a1-cd" column external_ids** B&R 2014-06-16 23:41 GMT+07:00 Eric Blake <eblake@redhat.com>:
On 06/16/2014 02:06 AM, Tuân Tạ Bá wrote:
Hi everyone,
I use KVM on Openstack Cloud. And KVM migrated successfull, but not ping.
*Source Host SIDE: *"/var/log/libvirt/libvirtd.log"
2014-06-16 05:15:50.599+0000: 2096: warning : qemuMigrationCancelDriveMirror:1383 : Unable to stop block job on drive-virtio-disk0 2014-06-16 05:15:50.601+0000: 2096: *warning : qemuMigrationCancelDriveMirror:1383 : Unable to stop block job on drive-virtio-disk1*
These warnings don't look good; have you investigated why they are occurring?
*2014-06-16 05:15:50.651+0000: 2096: error : virCommandWait:2348 : internal error: Child process (ovs-vsctl --timeout=5 get Interface tap6977e4a1-cd external_ids:PortData) unexpected exit status 1: ovs-vsctl: no key "PortData" in Interface record "tap6977e4a1-cd" column external_ids*
2014-06-16 05:15:50.651+0000: 2096: error : virNetDevOpenvswitchGetMigrateData:219 : Unable to run command to get OVS port data for interface tap6977e4a1-cd: Operation not permitted 2014-06-16 05:15:50.651+0000: 2096: error : qemuMigrationCookieNetworkAlloc:367 : Unable to run command to get OVS port data for interface tap6977e4a1-cd: Operation not permitted
Based on this failure, migration should not be successful.
2014-06-16 05:15:50.651+0000: 2096: warning : qemuMigrationRun:3262 : Unable to encode migration cookie 2014-06-16 05:15:53.321+0000: 2096: warning : virAuditSend:135 : Failed to send audit message virt=kvm op=stop reason=migrated vm="instance-0001b37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd vm-pid=-1: Operation not permitted
That doesn't look good either. Are you sure your system is set up correctly to allow audit messages to be generated? But it's probably unrelated to the failure you are seeing.
*Destination Host SIDE:* "/var/log/libvirt/libvirtd.log" ......
2014-06-16 05:15:41.419+0000: 2229: warning : virAuditSend:135 : Failed
to
send audit message virt=kvm resrc=cgroup reason=allow vm="instance-000 1b37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd cgroup="/sys/fs/cgroup/devices/machine/instance-0001b37c.libvirt-qemu/" class=major category=pty maj=88 acl=rw: Operation not permitted 2014-06-16 05:15:41.620+0000: 2229: warning : qemuDomainObjEnterMonitorInternal:1175 : This thread seems to be the async job owner; entering moni tor without asking for a nested job is dangerous 2014-06-16 05:15:43.552+0000: 2229: warning : virAuditSend:135 : Failed to send audit message virt=kvm resrc=vcpu reason=start vm="instance-0001b 37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd old-vcpu=0 new-vcpu=2: Operation not permitted 2014-06-16 05:15:43.552+0000: 2229: warning : virAuditSend:135 :* Failed to send audit message *virt=kvm op=start reason=migrated vm="instance-0001 b37c" uuid=631ac4c7-8e84-4cf6-ad98-93877cdbe1bd vm-pid=21063:* Operation not permitted*
1) I'm using libvirt version 1.1.1,
libvirt-bin 1.1.1-0ubuntu8.5~cloud0 libvirt0 1.1.1-0ubuntu8.5~cloud0 python-libvirt 1.1.1-0ubuntu8.5~cloud0
2) #cat /etc/libvirt/qemu.conf ... user = "root" group = "root"
I searched GOOGLE, René speaks "downgrading to libvirt 1.1.0 and reinstalling everything. They will ok". But I don't think that.
How to resolve this error?.
We've had some bug fixes for migration in the meantime; can you reproduce the error with libvirt 1.2.5?
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
-- Name: TA BA TUAN Work: Network System Administrator Company: Advanced Intelligence Technology Stock Company Mobile: 01638.369.743 Email: tuantb@ai-t.vn Adress: G Block, La Thanh Hotel, 218 Doi Can Str, Ba Dinh Dist, Hanoi, Vietnam
participants (2)
-
Eric Blake -
Tuân Tạ Bá