On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote:
On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
>
>>
>> But migration still fails
>>
>
> It seems like an unrelated failure. I do not know what's blocking
> migration traffic. Could you see if libvirtd.log and qemu logs at source
> and destinaiton have clues?
>
It seems that on VM.log under qemu of desdt host I have:
...
-incoming tcp:[::]:49153: Failed to bind socket: Address already in use
Is that port really taken (`ss -ntp` should tell by whom)?
See all:
- In libvirtd.log of source host
2013-10-07 23:20:54.471+0000: 1209: debug :
qemuMonitorOpenInternal:751 : QEMU_MONITOR_NEW: mon=0x7fc66412e820
refs=2 fd=30
2013-10-07 23:20:54.472+0000: 1209: warning :
qemuDomainObjEnterMonitorInternal:1136 : This thread seems to be the
async job owner; entering monitor without asking for a nested job is
dangerous
2013-10-07 23:20:54.472+0000: 1209: debug :
qemuMonitorSetCapabilities:1145 : mon=0x7fc66412e820
2013-10-07 23:20:54.472+0000: 1209: debug : qemuMonitorSend:887 :
QEMU_MONITOR_SEND_MSG: mon=0x7fc66412e820
msg={"execute":"qmp_capabilities","id":"libvirt-1"}
fd=-1
2013-10-07 23:20:54.769+0000: 1199: error : qemuMonitorIORead:505 :
Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.769+0000: 1199: debug : qemuMonitorIO:638 : Error
on monitor Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.769+0000: 1199: debug : qemuMonitorIO:672 :
Triggering error callback
2013-10-07 23:20:54.769+0000: 1199: debug :
qemuProcessHandleMonitorError:351 : Received error on 0x7fc664124fb0
'c8again32'
2013-10-07 23:20:54.769+0000: 1209: debug : qemuMonitorSend:899 : Send
command resulted in error Unable to read from monitor: Connection
reset by peer
2013-10-07 23:20:54.770+0000: 1199: debug : qemuMonitorIO:638 : Error
on monitor Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.770+0000: 1209: debug : virFileMakePathHelper:1283
: path=/var/run/libvirt/qemu mode=0777
2013-10-07 23:20:54.770+0000: 1199: debug : qemuMonitorIO:661 :
Triggering EOF callback
2013-10-07 23:20:54.770+0000: 1199: debug :
qemuProcessHandleMonitorEOF:294 : Received EOF on 0x7fc664124fb0
'c8again32'
2013-10-07 23:20:54.770+0000: 1209: debug : qemuProcessStop:3992 :
Shutting down VM 'c8again32' pid=18053 flags=0
2013-10-07 23:20:54.771+0000: 1209: error :
virNWFilterDHCPSnoopEnd:2135 : internal error ifname "vnet0" not in
key map
2013-10-07 23:20:54.782+0000: 1209: debug : virCommandRunAsync:2251 :
About to run /bin/sh -c 'IPT="/usr/sbin/iptables"
$IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
vnet0 -g FO-vnet0
$IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
$IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
$IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
$IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
$IPT -F FO-vnet0
$IPT -X FO-vnet0
$IPT -F FI-vnet0
$IPT -X FI-vnet0
$IPT -F HI-vnet0
$IPT -X HI-vnet0
IPT="/usr/sbin/ip6tables"
$IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
vnet0 -g FO-vnet0
$IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
$IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
$IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
$IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
$IPT -F FO-vnet0
$IPT -X FO-vnet0
$IPT -F FI-vnet0
$IPT -X FI-vnet0
$IPT -F HI-vnet0
$IPT -X HI-vnet0
EBT="/usr/sbin/ebtables"
$EBT -t nat -D PREROUTING -i vnet0 -j libvirt-I-vnet0
$EBT -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0
EBT="/usr/sbin/ebtables"
collect_chains()
{
for tmp2 in $*; do
for tmp in $($EBT -t nat -L $tmp2 | \
sed -n "/Bridge chain/,\$ s/.*-j \\([IO]-.*\\)/\\1/p");
do
echo $tmp
collect_chains $tmp
done
done
}
rm_chains()
{
for tmp in $*; do $EBT -t nat -F $tmp; done
for tmp in $*; do $EBT -t nat -X $tmp; done
}
tmp='\''
'\''
IFS='\'' '\'''\'' '\''$tmp
chains="$(collect_chains libvirt-I-vnet0 libvirt-O-vnet0)"
$EBT -t nat -F libvirt-I-vnet0
$EBT -t nat -F libvirt-O-vnet0
rm_chains $chains
$EBT -t nat -F libvirt-I-vnet0
$EBT -t nat -X libvirt-I-vnet0
$EBT -t nat -F libvirt-O-vnet0
$EBT -t nat -X libvirt-O-vnet0
'
2013-10-07 23:20:54.784+0000: 1209: debug : virCommandRunAsync:2256 :
Command result 0, with PID 18076
2013-10-07 23:20:54.863+0000: 1209: debug : virCommandRun:2125 :
Result exit status 255, stdout: '' stderr: 'iptables v1.4.18: goto
'FO-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'FO-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'FI-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'HI-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables: Bad rule (does a matching rule exist in that chain?).
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
ip6tables v1.4.18: goto 'FO-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables v1.4.18: goto 'FO-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables v1.4.18: goto 'FI-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables v1.4.18: goto 'HI-vnet0' is not a chain
Try `ip6tables -h' or 'ip6tables --help' for more information.
ip6tables: Bad rule (does a matching rule exist in that chain?).
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
ip6tables: No chain/target/match by that name.
Illegal target name 'libvirt-O-vnet0'.
Chain 'libvirt-O-vnet0' doesn't exist.
Chain 'libvirt-O-vnet0' doesn't exist.
Chain 'libvirt-O-vnet0' doesn't exist.
Chain 'libvirt-O-vnet0' doesn't exist.
'
2013-10-07 23:20:54.863+0000: 1209: debug : qemuMonitorClose:821 :
QEMU_MONITOR_CLOSE: mon=0x7fc66412e820 refs=2
2013-10-07 23:20:54.863+0000: 1209: debug : qemuProcessKill:3951 :
vm=c8again32 pid=18053 flags=5
2013-10-07 23:20:54.863+0000: 1209: debug :
virProcessKillPainfully:269 : vpid=18053 force=1
2013-10-07 23:20:54.863+0000: 1209: debug : qemuDomainCleanupRun:2132
: driver=0x7fc664024cd0, vm=c8again32
2013-10-07 23:20:54.863+0000: 1209: debug :
qemuProcessAutoDestroyRemove:4504 : vm=c8again32
2013-10-07 23:20:54.863+0000: 1209: debug :
virQEMUCloseCallbacksUnset:744 : vm=c8again32,
uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, cb=0x7fc66b6fe570
2013-10-07 23:20:54.864+0000: 1209: error :
virPortAllocatorRelease:174 : port 0 must be in range (5900, 65535)
2013-10-07 23:20:54.865+0000: 1209: debug : qemuDomainObjEndJob:1070 :
Stopping job: none (async=migration in)
2013-10-07 23:20:54.865+0000: 1209: debug :
qemuDomainObjEndAsyncJob:1088 : Stopping async job: migration in
2013-10-07 23:20:54.865+0000: 1199: debug :
qemuProcessHandleMonitorEOF:306 : Domain 0x7fc664124fb0 is not active,
ignoring EOF
Caught Segmentation violation dumping internal log buffer:
This line one seems ominous. Can libvir-list help with it?