Re: [libvirt] [libvirt-users] Help? Running into problems with migrateToURI2() and virDomainDefCheckABIStability()

12 Feb 2014

      On 02/12/2014 01:03 PM, Chris Friesen wrote:
...
On 02/11/2014 04:45 PM, Cole Robinson wrote:
...
On 02/10/2014 06:46 PM, Chris Friesen wrote:
...
Hi,
We've run into a problem with libvirt 1.1.2 and are looking for some comments
on whether this is a bug or design intent.
We're trying to use migrateToURI() but we're using a few things (numatune,
vcpu mask, etc.) that may need adjustment during the migration.  We found that
migrateToURI2() mostly works if we use XML created by copying the domain XML
from the running instance and modifying the appropriate sections.
The problem that we're seeing is that the serial console checking in
libvirt/src/conf/domain_conf.c::virDomainDefCheckABIStability() is failing
even though we haven't touched the serial console XML:
...
I think we will have to see the XML your passing before and after edit to
figure out what's going wrong. Also, showing all the arguments you are
invoking the migration APIs with.
Okay, this is going to be a long message.  :)
We're calling it as:
dom.migrateToURI2(duri, muri, xml_str, logical_sum, None, CONF.live_migration_bandwidth)
Here are the supplied arguments:
duri = u'qemu+tcp://compute-0/system'
muri = u'tcp://compute-0'
flaglist = ['VIR_MIGRATE_UNDEFINE_SOURCE', 'VIR_MIGRATE_PEER2PEER', 'VIR_MIGRATE_LIVE']
logical_sum = 19,  (logical OR of flaglist)
CONF.live_migration_bandwidth = 0
Grabbing original XML prior to migration:
dom_xml = etree.fromstring(dom.XMLDesc(0))
xml_str = etree.tostring(dom_xml)
xml_str (BEFORE) =
<domain type="qemu" id="2">
  <name>instance-00000006</name>
  <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid>
  <dpdk>
    <process type="secondary"/>
    <file prefix="vs"/>
    <cpu list="0"/>
    <memory channels="4"/>
  </dpdk>
  <memory unit="KiB">262144</memory>
  <currentMemory unit="KiB">262144</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static" cpuset="1-2">2</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="1"/>
    <vcpupin vcpu="1" cpuset="2"/>
    <emulatorpin cpuset="1-2"/>
  </cputune>
  <numatune>
    <memory mode="preferred" nodeset="0"/>
  </numatune>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2013.2.2</entry>
      <entry name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry>
      <entry name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry>
    </system>
  </sysinfo>
  <os>
    <type arch="x86_64" machine="pc-i440fx-1.4">hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/>
      <target dev="vda" bus="virtio"/>
      <alias name="virtio-disk0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </disk>
    <controller type="usb" index="0">
      <alias name="usb0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pci-root">
      <alias name="pci.0"/>
    </controller>
    <interface type="vswitch">
      <mac address="fa:16:3e:fb:34:6b"/>
      <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/>
      <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/>
      <model type="e1000"/>
      <alias name="net0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </interface>
    <interface type="vswitch">
      <mac address="fa:16:3e:d5:b9:50"/>
      <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/>
      <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/>
      <model type="e1000"/>
      <alias name="net1"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </interface>
    <serial type="file">
      <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
    <input type="tablet" bus="usb">
      <alias name="input0"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0" keymap="en-us">
      <listen type="address" address="0.0.0.0"/>
    </graphics>
    <video>
      <model type="cirrus" vram="9216" heads="1"/>
      <alias name="video0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </video>
    <memballoon model="virtio">
      <alias name="balloon0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </memballoon>
  </devices>
  <seclabel type="none"/>
</domain>
This is the modified XML, specifically updating vcpu, cputune, numatune.
In this case they actually have the same values as above but there are
whitespace differences:
xml_str (MODIFIED) =
<domain type="qemu" id="2">
  <name>instance-00000006</name>
  <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid>
  <dpdk>
    <process type="secondary"/>
    <file prefix="vs"/>
    <cpu list="0"/>
    <memory channels="4"/>
  </dpdk>
  <memory unit="KiB">262144</memory>
  <currentMemory unit="KiB">262144</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static" cpuset="1,2">2</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="1"/><vcpupin vcpu="1" cpuset="2"/><emulatorpin cpuset="1,2"/></cputune>
  <numatune>
    <memory mode="preferred" nodeset="0"/></numatune>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2013.2.2</entry>
      <entry name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry>
      <entry name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry>
    </system>
  </sysinfo>
  <os>
    <type arch="x86_64" machine="pc-i440fx-1.4">hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/>
      <target dev="vda" bus="virtio"/>
      <alias name="virtio-disk0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </disk>
    <controller type="usb" index="0">
      <alias name="usb0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pci-root">
      <alias name="pci.0"/>
    </controller>
    <interface type="vswitch">
      <mac address="fa:16:3e:fb:34:6b"/>
      <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/>
      <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/>
      <model type="e1000"/>
      <alias name="net0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </interface>
    <interface type="vswitch">
      <mac address="fa:16:3e:d5:b9:50"/>
      <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/>
      <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/>
      <model type="e1000"/>
      <alias name="net1"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </interface>
    <serial type="file">
      <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
    <input type="tablet" bus="usb">
      <alias name="input0"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0" keymap="en-us">
      <listen type="address" address="0.0.0.0"/>
    </graphics>
    <video>
      <model type="cirrus" vram="9216" heads="1"/>
      <alias name="video0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </video>
    <memballoon model="virtio">
      <alias name="balloon0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </memballoon>
  </devices>
  <seclabel type="none"/>
</domain>
Calling migrateToURI2 then gives the following error :
2014-02-12 03:37:17.738 1251 ERROR nova.virt.libvirt.driver [-] [instance: 2e13869b-edc0-40dc-92e3-1f2ffde60cce] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0
If we remove the this section :
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
Then we get the following error :
2014-02-12 04:19:12.689 1278 ERROR nova.virt.libvirt.driver [-] [instance: 90335c32-a7ae-4c99-8e5a-cd177f6b3c22] Live Migration failure: unsupported configuration: Target domain serial port count 1 does not match source 2
If we remove just the following line:
      <source path="/dev/pts/1"/>
so the modified portion looks like this,
    <serial type="file">
      <source path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
    <input type="tablet" bus="usb">
Then we get this error :
2014-02-12 04:29:06.402 5175 ERROR nova.virt.libvirt.driver [-] [instance: 00b99f94-436e-406b-8459-83e77758f154] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0
If we modify it with empty <serial type='pty'> element like this:
<serial type="file">
      <source path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      </serial>
    <console type="file">
      <source path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
Then we get this error:
2014-02-12 04:36:45.686 5358 ERROR nova.virt.libvirt.driver [-] [instance: b1103349-8c9f-4315-9144-03fc22f09be1] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0
If we modify it with empty <serial type='pty'> element and also remove
<console type='file'> section completely like this :
<serial type="file">
      <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
    </serial>
Then this successfully migrates!
However, it then fails on subsequent migration back to the original host.  Prior to migrating
back the domain XML serial/console section looks like:
<serial type="file">
      <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
After modifying empty <serial type='pty'> element and removing the <console type='file'> section
it looks like this :
<serial type="file">
      <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
    </serial>
But this give the following error when migrating back:
2014-02-12 06:00:19.831 7125 ERROR nova.virt.libvirt.driver [-] [instance: 8cccb4ad-2abc-4513-b8fd-bfb811301677] Live Migration failure: unsupported configuration: Target domain console count 0 does not match source 1
Thanks for the detailed info. Can you put that into an upstream libvirt bug
report? http://libvirt.org/bugs.html

Does this sound familiar to any other libvirt devs?

- Cole

Cole Robinson

tags

participants (1)