12:35 p.m.
On 02/12/2014 01:03 PM, Chris Friesen wrote:
On 02/11/2014 04:45 PM, Cole Robinson wrote:
> On 02/10/2014 06:46 PM, Chris Friesen wrote:
>> Hi,
>>
>> We've run into a problem with libvirt 1.1.2 and are looking for some
comments
>> on whether this is a bug or design intent.
>>
>> We're trying to use migrateToURI() but we're using a few things
(numatune,
>> vcpu mask, etc.) that may need adjustment during the migration. We found that
>> migrateToURI2() mostly works if we use XML created by copying the domain XML
>> from the running instance and modifying the appropriate sections.
>>
>> The problem that we're seeing is that the serial console checking in
>> libvirt/src/conf/domain_conf.c::virDomainDefCheckABIStability() is failing
>> even though we haven't touched the serial console XML:
> I think we will have to see the XML your passing before and after edit to
> figure out what's going wrong. Also, showing all the arguments you are
> invoking the migration APIs with.
Okay, this is going to be a long message. :)
We're calling it as:
dom.migrateToURI2(duri, muri, xml_str, logical_sum, None, CONF.live_migration_bandwidth)
Here are the supplied arguments:
duri = u'qemu+tcp://compute-0/system'
muri = u'tcp://compute-0'
flaglist = ['VIR_MIGRATE_UNDEFINE_SOURCE', 'VIR_MIGRATE_PEER2PEER',
'VIR_MIGRATE_LIVE']
logical_sum = 19, (logical OR of flaglist)
CONF.live_migration_bandwidth = 0
Grabbing original XML prior to migration:
dom_xml = etree.fromstring(dom.XMLDesc(0))
xml_str = etree.tostring(dom_xml)
xml_str (BEFORE) =
<domain type="qemu" id="2">
<name>instance-00000006</name>
<uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid>
<dpdk>
<process type="secondary"/>
<file prefix="vs"/>
<cpu list="0"/>
<memory channels="4"/>
</dpdk>
<memory unit="KiB">262144</memory>
<currentMemory unit="KiB">262144</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement="static" cpuset="1-2">2</vcpu>
<cputune>
<vcpupin vcpu="0" cpuset="1"/>
<vcpupin vcpu="1" cpuset="2"/>
<emulatorpin cpuset="1-2"/>
</cputune>
<numatune>
<memory mode="preferred" nodeset="0"/>
</numatune>
<sysinfo type="smbios">
<system>
<entry name="manufacturer">OpenStack Foundation</entry>
<entry name="product">OpenStack Nova</entry>
<entry name="version">2013.2.2</entry>
<entry
name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry>
<entry
name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry>
</system>
</sysinfo>
<os>
<type arch="x86_64"
machine="pc-i440fx-1.4">hvm</type>
<boot dev="hd"/>
<smbios mode="sysinfo"/>
</os>
<features>
<acpi/>
<apic/>
</features>
<clock offset="utc"/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"
cache="none"/>
<source
file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/>
<target dev="vda" bus="virtio"/>
<alias name="virtio-disk0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x05" function="0x0"/>
</disk>
<controller type="usb" index="0">
<alias name="usb0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x01" function="0x2"/>
</controller>
<controller type="pci" index="0"
model="pci-root">
<alias name="pci.0"/>
</controller>
<interface type="vswitch">
<mac address="fa:16:3e:fb:34:6b"/>
<source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/>
<target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/>
<model type="e1000"/>
<alias name="net0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x03" function="0x0"/>
</interface>
<interface type="vswitch">
<mac address="fa:16:3e:d5:b9:50"/>
<source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/>
<target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/>
<model type="e1000"/>
<alias name="net1"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x04" function="0x0"/>
</interface>
<serial type="file">
<source
path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
<target port="0"/>
<alias name="serial0"/>
</serial>
<serial type="pty">
<source path="/dev/pts/1"/>
<target port="1"/>
<alias name="serial1"/>
</serial>
<console type="file">
<source
path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
<target type="serial" port="0"/>
<alias name="serial0"/>
</console>
<input type="tablet" bus="usb">
<alias name="input0"/>
</input>
<input type="mouse" bus="ps2"/>
<graphics type="vnc" port="5900" autoport="yes"
listen="0.0.0.0" keymap="en-us">
<listen type="address" address="0.0.0.0"/>
</graphics>
<video>
<model type="cirrus" vram="9216" heads="1"/>
<alias name="video0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x02" function="0x0"/>
</video>
<memballoon model="virtio">
<alias name="balloon0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x06" function="0x0"/>
</memballoon>
</devices>
<seclabel type="none"/>
</domain>
This is the modified XML, specifically updating vcpu, cputune, numatune.
In this case they actually have the same values as above but there are
whitespace differences:
xml_str (MODIFIED) =
<domain type="qemu" id="2">
<name>instance-00000006</name>
<uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid>
<dpdk>
<process type="secondary"/>
<file prefix="vs"/>
<cpu list="0"/>
<memory channels="4"/>
</dpdk>
<memory unit="KiB">262144</memory>
<currentMemory unit="KiB">262144</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement="static" cpuset="1,2">2</vcpu>
<cputune>
<vcpupin vcpu="0" cpuset="1"/><vcpupin
vcpu="1" cpuset="2"/><emulatorpin
cpuset="1,2"/></cputune>
<numatune>
<memory mode="preferred" nodeset="0"/></numatune>
<sysinfo type="smbios">
<system>
<entry name="manufacturer">OpenStack Foundation</entry>
<entry name="product">OpenStack Nova</entry>
<entry name="version">2013.2.2</entry>
<entry
name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry>
<entry
name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry>
</system>
</sysinfo>
<os>
<type arch="x86_64"
machine="pc-i440fx-1.4">hvm</type>
<boot dev="hd"/>
<smbios mode="sysinfo"/>
</os>
<features>
<acpi/>
<apic/>
</features>
<clock offset="utc"/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"
cache="none"/>
<source
file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/>
<target dev="vda" bus="virtio"/>
<alias name="virtio-disk0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x05" function="0x0"/>
</disk>
<controller type="usb" index="0">
<alias name="usb0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x01" function="0x2"/>
</controller>
<controller type="pci" index="0"
model="pci-root">
<alias name="pci.0"/>
</controller>
<interface type="vswitch">
<mac address="fa:16:3e:fb:34:6b"/>
<source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/>
<target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/>
<model type="e1000"/>
<alias name="net0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x03" function="0x0"/>
</interface>
<interface type="vswitch">
<mac address="fa:16:3e:d5:b9:50"/>
<source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/>
<target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/>
<model type="e1000"/>
<alias name="net1"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x04" function="0x0"/>
</interface>
<serial type="file">
<source
path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
<target port="0"/>
<alias name="serial0"/>
</serial>
<serial type="pty">
<source path="/dev/pts/1"/>
<target port="1"/>
<alias name="serial1"/>
</serial>
<console type="file">
<source
path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
<target type="serial" port="0"/>
<alias name="serial0"/>
</console>
<input type="tablet" bus="usb">
<alias name="input0"/>
</input>
<input type="mouse" bus="ps2"/>
<graphics type="vnc" port="5900" autoport="yes"
listen="0.0.0.0" keymap="en-us">
<listen type="address" address="0.0.0.0"/>
</graphics>
<video>
<model type="cirrus" vram="9216" heads="1"/>
<alias name="video0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x02" function="0x0"/>
</video>
<memballoon model="virtio">
<alias name="balloon0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x06" function="0x0"/>
</memballoon>
</devices>
<seclabel type="none"/>
</domain>
Calling migrateToURI2 then gives the following error :
2014-02-12 03:37:17.738 1251 ERROR nova.virt.libvirt.driver [-] [instance:
2e13869b-edc0-40dc-92e3-1f2ffde60cce] Live Migration failure: unsupported configuration:
Target domain console count 1 does not match source 0
If we remove the this section :
<serial type="pty">
<source path="/dev/pts/1"/>
<target port="1"/>
<alias name="serial1"/>
</serial>
Then we get the following error :
2014-02-12 04:19:12.689 1278 ERROR nova.virt.libvirt.driver [-] [instance:
90335c32-a7ae-4c99-8e5a-cd177f6b3c22] Live Migration failure: unsupported configuration:
Target domain serial port count 1 does not match source 2
If we remove just the following line:
<source path="/dev/pts/1"/>
so the modified portion looks like this,
<serial type="file">
<source
path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/>
<target port="0"/>
<alias name="serial0"/>
</serial>
<serial type="pty">
<target port="1"/>
<alias name="serial1"/>
</serial>
<console type="file">
<source
path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/>
<target type="serial" port="0"/>
<alias name="serial0"/>
</console>
<input type="tablet" bus="usb">
Then we get this error :
2014-02-12 04:29:06.402 5175 ERROR nova.virt.libvirt.driver [-] [instance:
00b99f94-436e-406b-8459-83e77758f154] Live Migration failure: unsupported configuration:
Target domain console count 1 does not match source 0
If we modify it with empty <serial type='pty'> element like this:
<serial type="file">
<source
path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/>
<target port="0"/>
<alias name="serial0"/>
</serial>
<serial type="pty">
</serial>
<console type="file">
<source
path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/>
<target type="serial" port="0"/>
<alias name="serial0"/>
</console>
Then we get this error:
2014-02-12 04:36:45.686 5358 ERROR nova.virt.libvirt.driver [-] [instance:
b1103349-8c9f-4315-9144-03fc22f09be1] Live Migration failure: unsupported configuration:
Target domain console count 1 does not match source 0
If we modify it with empty <serial type='pty'> element and also remove
<console type='file'> section completely like this :
<serial type="file">
<source
path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
<target port="0"/>
<alias name="serial0"/>
</serial>
<serial type="pty">
</serial>
Then this successfully migrates!
However, it then fails on subsequent migration back to the original host. Prior to
migrating
back the domain XML serial/console section looks like:
<serial type="file">
<source
path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
<target port="0"/>
<alias name="serial0"/>
</serial>
<serial type="pty">
<source path="/dev/pts/1"/>
<target port="1"/>
<alias name="serial1"/>
</serial>
<console type="file">
<source
path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
<target type="serial" port="0"/>
<alias name="serial0"/>
</console>
After modifying empty <serial type='pty'> element and removing the
<console type='file'> section
it looks like this :
<serial type="file">
<source
path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
<target port="0"/>
<alias name="serial0"/>
</serial>
<serial type="pty">
</serial>
But this give the following error when migrating back:
2014-02-12 06:00:19.831 7125 ERROR nova.virt.libvirt.driver [-] [instance:
8cccb4ad-2abc-4513-b8fd-bfb811301677] Live Migration failure: unsupported configuration:
Target domain console count 0 does not match source 1
Thanks for the detailed info. Can you put that into an upstream libvirt bug
report? http://libvirt.org/bugs.html
Does this sound familiar to any other libvirt devs?
- Cole