On 02/11/2014 04:45 PM, Cole Robinson wrote: > On 02/10/2014 06:46 PM, Chris Friesen wrote: >> Hi, >> >> We've run into a problem with libvirt 1.1.2 and are looking for some comments >> on whether this is a bug or design intent. >> >> We're trying to use migrateToURI() but we're using a few things (numatune, >> vcpu mask, etc.) that may need adjustment during the migration. We found that >> migrateToURI2() mostly works if we use XML created by copying the domain XML >> from the running instance and modifying the appropriate sections. >> >> The problem that we're seeing is that the serial console checking in >> libvirt/src/conf/domain_conf.c::virDomainDefCheckABIStability() is failing >> even though we haven't touched the serial console XML: > I think we will have to see the XML your passing before and after edit to > figure out what's going wrong. Also, showing all the arguments you are > invoking the migration APIs with. Okay, this is going to be a long message. :) We're calling it as: dom.migrateToURI2(duri, muri, xml_str, logical_sum, None, CONF.live_migration_bandwidth) Here are the supplied arguments: duri = u'qemu+tcp://compute-0/system' muri = u'tcp://compute-0' flaglist = ['VIR_MIGRATE_UNDEFINE_SOURCE', 'VIR_MIGRATE_PEER2PEER', 'VIR_MIGRATE_LIVE'] logical_sum = 19, (logical OR of flaglist) CONF.live_migration_bandwidth = 0 Grabbing original XML prior to migration: dom_xml = etree.fromstring(dom.XMLDesc(0)) xml_str = etree.tostring(dom_xml) xml_str (BEFORE) = <domain type="qemu" id="2"> <name>instance-00000006</name> <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid> <dpdk> <process type="secondary"/> <file prefix="vs"/> <cpu list="0"/> <memory channels="4"/> </dpdk> <memory unit="KiB">262144</memory> <currentMemory unit="KiB">262144</currentMemory> <memoryBacking> <hugepages/> </memoryBacking> <vcpu placement="static" cpuset="1-2">2</vcpu> <cputune> <vcpupin vcpu="0" cpuset="1"/> <vcpupin vcpu="1" cpuset="2"/> <emulatorpin cpuset="1-2"/> </cputune> <numatune> <memory mode="preferred" nodeset="0"/> </numatune> <sysinfo type="smbios"> <system> <entry name="manufacturer">OpenStack Foundation</entry> <entry name="product">OpenStack Nova</entry> <entry name="version">2013.2.2</entry> <entry name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry> <entry name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry> </system> </sysinfo> <os> <type arch="x86_64" machine="pc-i440fx-1.4">hvm</type> <boot dev="hd"/> <smbios mode="sysinfo"/> </os> <features> <acpi/> <apic/> </features> <clock offset="utc"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type="file" device="disk"> <driver name="qemu" type="qcow2" cache="none"/> <source file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/> <target dev="vda" bus="virtio"/> <alias name="virtio-disk0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/> </disk> <controller type="usb" index="0"> <alias name="usb0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/> </controller> <controller type="pci" index="0" model="pci-root"> <alias name="pci.0"/> </controller> <interface type="vswitch"> <mac address="fa:16:3e:fb:34:6b"/> <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/> <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/> <model type="e1000"/> <alias name="net0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/> </interface> <interface type="vswitch"> <mac address="fa:16:3e:d5:b9:50"/> <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/> <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/> <model type="e1000"/> <alias name="net1"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/> </interface> <serial type="file"> <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/> <target port="0"/> <alias name="serial0"/> </serial> <serial type="pty"> <source path="/dev/pts/1"/> <target port="1"/> <alias name="serial1"/> </serial> <console type="file"> <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/> <target type="serial" port="0"/> <alias name="serial0"/> </console> <input type="tablet" bus="usb"> <alias name="input0"/> </input> <input type="mouse" bus="ps2"/> <graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0" keymap="en-us"> <listen type="address" address="0.0.0.0"/> </graphics> <video> <model type="cirrus" vram="9216" heads="1"/> <alias name="video0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/> </video> <memballoon model="virtio"> <alias name="balloon0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/> </memballoon> </devices> <seclabel type="none"/> </domain> This is the modified XML, specifically updating vcpu, cputune, numatune. In this case they actually have the same values as above but there are whitespace differences: xml_str (MODIFIED) = <domain type="qemu" id="2"> <name>instance-00000006</name> <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid> <dpdk> <process type="secondary"/> <file prefix="vs"/> <cpu list="0"/> <memory channels="4"/> </dpdk> <memory unit="KiB">262144</memory> <currentMemory unit="KiB">262144</currentMemory> <memoryBacking> <hugepages/> </memoryBacking> <vcpu placement="static" cpuset="1,2">2</vcpu> <cputune> <vcpupin vcpu="0" cpuset="1"/><vcpupin vcpu="1" cpuset="2"/><emulatorpin cpuset="1,2"/></cputune> <numatune> <memory mode="preferred" nodeset="0"/></numatune> <sysinfo type="smbios"> <system> <entry name="manufacturer">OpenStack Foundation</entry> <entry name="product">OpenStack Nova</entry> <entry name="version">2013.2.2</entry> <entry name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry> <entry name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry> </system> </sysinfo> <os> <type arch="x86_64" machine="pc-i440fx-1.4">hvm</type> <boot dev="hd"/> <smbios mode="sysinfo"/> </os> <features> <acpi/> <apic/> </features> <clock offset="utc"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type="file" device="disk"> <driver name="qemu" type="qcow2" cache="none"/> <source file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/> <target dev="vda" bus="virtio"/> <alias name="virtio-disk0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/> </disk> <controller type="usb" index="0"> <alias name="usb0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/> </controller> <controller type="pci" index="0" model="pci-root"> <alias name="pci.0"/> </controller> <interface type="vswitch"> <mac address="fa:16:3e:fb:34:6b"/> <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/> <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/> <model type="e1000"/> <alias name="net0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/> </interface> <interface type="vswitch"> <mac address="fa:16:3e:d5:b9:50"/> <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/> <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/> <model type="e1000"/> <alias name="net1"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/> </interface> <serial type="file"> <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/> <target port="0"/> <alias name="serial0"/> </serial> <serial type="pty"> <source path="/dev/pts/1"/> <target port="1"/> <alias name="serial1"/> </serial> <console type="file"> <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/> <target type="serial" port="0"/> <alias name="serial0"/> </console> <input type="tablet" bus="usb"> <alias name="input0"/> </input> <input type="mouse" bus="ps2"/> <graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0" keymap="en-us"> <listen type="address" address="0.0.0.0"/> </graphics> <video> <model type="cirrus" vram="9216" heads="1"/> <alias name="video0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/> </video> <memballoon model="virtio"> <alias name="balloon0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/> </memballoon> </devices> <seclabel type="none"/> </domain> Calling migrateToURI2 then gives the following error : 2014-02-12 03:37:17.738 1251 ERROR nova.virt.libvirt.driver [-] [instance: 2e13869b-edc0-40dc-92e3-1f2ffde60cce] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0 If we remove the this section : <serial type="pty"> <source path="/dev/pts/1"/> <target port="1"/> <alias name="serial1"/> </serial> Then we get the following error : 2014-02-12 04:19:12.689 1278 ERROR nova.virt.libvirt.driver [-] [instance: 90335c32-a7ae-4c99-8e5a-cd177f6b3c22] Live Migration failure: unsupported configuration: Target domain serial port count 1 does not match source 2 If we remove just the following line: <source path="/dev/pts/1"/> so the modified portion looks like this, <serial type="file"> <source path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/> <target port="0"/> <alias name="serial0"/> </serial> <serial type="pty"> <target port="1"/> <alias name="serial1"/> </serial> <console type="file"> <source path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/> <target type="serial" port="0"/> <alias name="serial0"/> </console> <input type="tablet" bus="usb"> Then we get this error : 2014-02-12 04:29:06.402 5175 ERROR nova.virt.libvirt.driver [-] [instance: 00b99f94-436e-406b-8459-83e77758f154] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0 If we modify it with empty <serial type='pty'> element like this: <serial type="file"> <source path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/> <target port="0"/> <alias name="serial0"/> </serial> <serial type="pty"> </serial> <console type="file"> <source path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/> <target type="serial" port="0"/> <alias name="serial0"/> </console> Then we get this error: 2014-02-12 04:36:45.686 5358 ERROR nova.virt.libvirt.driver [-] [instance: b1103349-8c9f-4315-9144-03fc22f09be1] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0 If we modify it with empty <serial type='pty'> element and also remove <console type='file'> section completely like this : <serial type="file"> <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/> <target port="0"/> <alias name="serial0"/> </serial> <serial type="pty"> </serial> Then this successfully migrates! However, it then fails on subsequent migration back to the original host. Prior to migrating back the domain XML serial/console section looks like: <serial type="file"> <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/> <target port="0"/> <alias name="serial0"/> </serial> <serial type="pty"> <source path="/dev/pts/1"/> <target port="1"/> <alias name="serial1"/> </serial> <console type="file"> <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/> <target type="serial" port="0"/> <alias name="serial0"/> </console> After modifying empty <serial type='pty'> element and removing the <console type='file'> section it looks like this : <serial type="file"> <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/> <target port="0"/> <alias name="serial0"/> </serial> <serial type="pty"> </serial> But this give the following error when migrating back: 2014-02-12 06:00:19.831 7125 ERROR nova.virt.libvirt.driver [-] [instance: 8cccb4ad-2abc-4513-b8fd-bfb811301677] Live Migration failure: unsupported configuration: Target domain console count 0 does not match source 1