On Mon, Mar 03, 2014 at 02:15:43PM +0000, Struan Bartlett wrote:

On 03/03/2014 13:42, Martin Kletzander wrote:

On Mon, Mar 03, 2014 at 11:15:51AM +0000, Struan Bartlett wrote:

On 03/03/2014 10:55, Martin Kletzander wrote:

On Mon, Mar 03, 2014 at 10:47:03AM +0000, Struan Bartlett wrote:

On 03/03/2014 10:44, Martin Kletzander wrote:

On Mon, Mar 03, 2014 at 10:30:11AM +0000, Struan Bartlett wrote:

Hi Martin

Thanks for your response. Here's the output of that grep:

# grep ^flags /proc/cpuinfo | sort -u
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall
nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor
ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2
x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat xsaveopt
pln pts dtherm tpr_shadow vnmi flexpriority ept vpid

I've just managed to install the libvirt-bin:amd64 package on the same
machine, on the wheezy-i386 distribution. The output of 'virsh
capabilities' is now reported correctly, and my VMs that required
SandyBridge are now booting.

Have you any further suggestions?

Oh, I missed the fact that it was 32 bit distro.  In that case, I
guess some cpu features won't be available.  No other ideas.

Ok thanks Martin. If the amd64 and i386 versions of the package are
behaving differently in this respect, on otherwise the same hardware,
kernel and distro, then I can only presume this is a bug, so my next
step would be to raise this on the development list.

Just a guess, but our CPUID instruction might be translated to
assembly code differently.  You can try running 'cpuid -ir' (or 'cpuid
-ir1' if it's the same on all cpus) on both machines, that will give
you the results libvirt is processing.  If that output is different
than there is nothing libvirt can do.  If it's the same, it might be a
libvirt bug.

You might be onto something here. The following are run on the same
physical machine:

# apt-get install cpuid:i386
# cpuid -ir1 >/tmp/cpuid-ir1-i386
# apt-get install cpuid:amd64
# cpuid -ir1 >/tmp/cpuid-ir1-amd64
# diff -ubw /tmp/cpuid-ir1-i386 /tmp/cpuid-ir1-amd64
--- /tmp/cpuid-ir1-i386    2014-03-03 11:13:00.839208222 +0000
+++ /tmp/cpuid-ir1-amd64    2014-03-03 11:13:13.283307377 +0000
@@ -10,11 +10,11 @@
   00000008 00000000 00000000 00000000 00000000
   00000009 00000000 00000000 00000000 00000000
   0000000a 07300403 00000000 00000000 00000603
-0000000b 00000000 00000000 000000e8 0000002c
+0000000b 00000000 00000000 0000006f 0000002c

This one has *probably* no significance in libvirt code since we only
call cpuid functions 0x00000001, 0x00000007 and 0x80000001 (which is
enough for us to determine all we need).

   0000000c 00000000 00000000 00000000 00000000
   0000000d 00000000 00000000 00000000 00000000
   80000000 80000008 00000000 00000000 00000000
-80000001 00000000 00000000 00000001 2c100000
+80000001 00000000 00000000 00000001 2c100800

But this is something.

This means in second run the CPU reports it has "syscall" and in the
first one it doesn't.  Since "syscall" is x86_64 instruction used
instead of i386's "int 0x80" (IIRC) it makes sense that it is missing
on 32-bit CPU (when the code is running in 32-bit mode).

   80000002 20202020 6e492020 286c6574 58202952
   80000003 286e6f65 43202952 45205550 36322d35
   80000004 204c3035 20402030 30382e31 007a4847

Could this be enough difference to throw cpu-detection off?

Even one bit is enough if that bit represents a flag which is not in
the CPU you are looking for (i.e. SandyBridge).

I must admin I was too lazy to compare the differences between the
processor definitions you've sent (basically because I'm fairly
certain we have no such mistakes in our code O:-) ).

But I did that now and indeed, you can see that the difference is only
in the "syscall" feature.  You can do that yourselves using the info
from "cpu_map.xml" if you don't believe me ;)

Martin

Martin

Many thanks for helping me get to the bottom of this, though could you
clarify what you mean when you say that the difference is only in the
"syscall" feature? I've looked at cpu_map.xml, and compared the features
for n270 (pentiumpro+coreduo+ssse3) with SandyBridge
(pentiumpro+Conroe+Penryn+Nehalem+Westmere+pclmuldq+SandyBridge) and am
seeing quite a few differences (see below). The main issue though for

You have to also add the features that libvirt added in the output of
'virsh capabilities' and then try to match the differences.  I did it
by hand, so there might be a mistake or two, but not this much.  Add
the features from 'virsh capabilities' reported with x86_64 libvirt to
the SandyBridge-features and the features from i386 libvirt into
n270-features and you'll see.

me, is that the libvirt architecture need not be representative of the
virtualisation architecture. Apart from the issue of CPU detection, a
user should otherwise be fine running libvirt-bin:i386 with
qemu-system-x86_64:amd64 (which is our aim, leveraging Debian Multiarch

It... should, yes.  But given the current implementation there is
nothing we can do about this.  There's huge work being done by Jiri
(from libvirt) and Eduardo (from qemu) making libvirt more aware of
qemu CPUs.  That should help in the future, but since they are
experiencing many problems (and I'm considering only those which I
know of, I'm sure they have more of them) it won't take place in very
near future (just my guess).

to run only the 64bit binaries that really need 64bit capability or
large RAM, while keeping the memory footprint of other binaries small).
Is there a workaround? Just customise the cpu_map.xml file?

You *should not* customize that.  libvirt might suggest different
CPUs, it will get overridden with updates etc.  However, I cannot
force you to keep the file that way...

How about trying out the x32 ABI [1] [2]?  Does Debian make something
like even possible?

Martin

[1] http://en.wikipedia.org/wiki/X32_ABI
[2] https://sites.google.com/site/x32abi/

# diff -ubw /tmp/n270-features /tmp/SandyBridge-features
--- /tmp/n270-features    2014-03-03 14:05:02.003880511 +0000
+++ /tmp/SandyBridge-features    2014-03-03 14:04:56.063834563 +0000
@@ -1,25 +1,38 @@
+      <feature name='aes'/>
        <feature name='apic'/>
+      <feature name='avx'/>
        <feature name='clflush'/>
        <feature name='cmov'/>
+      <feature name='cx16'/>
        <feature name='cx8'/>
        <feature name='de'/>
        <feature name='fpu'/>
        <feature name='fxsr'/>
+      <feature name='lahf_lm'/>
+      <feature name='lm'/>
        <feature name='mca'/>
        <feature name='mce'/>
        <feature name='mmx'/>
-      <feature name='monitor'/>
        <feature name='msr'/>
        <feature name='mtrr'/>
        <feature name='nx'/>
        <feature name='pae'/>
        <feature name='pat'/>
+      <feature name='pclmuldq'/>
        <feature name='pge'/>
        <feature name='pni'/>
+      <feature name='popcnt'/>
        <feature name='pse'/>
+      <feature name='pse36'/>
+      <feature name='rdtscp'/>
        <feature name='sep'/>
        <feature name='sse'/>
        <feature name='sse2'/>
+      <feature name='sse4.1'/>
+      <feature name='sse4.2'/>
        <feature name='ssse3'/>
+      <feature name='syscall'/>
        <feature name='tsc'/>
-      <feature name='vme'/>
+      <feature name='tsc-deadline'/>
+      <feature name='x2apic'/>
+      <feature name='xsave'/>

Struan

Martin

Struan

On 03/03/2014 10:00, Martin Kletzander wrote:

On Fri, Feb 28, 2014 at 03:45:01PM +0000, Struan Bartlett wrote:

Hi

On a range of Dell servers containing Intel 64bit processors, 'virsh
capabilities' reports the cpu differently on Debian Wheezy-amd64 and
Wheezy-i386. The results given by the Wheezy-i386 version seem very
wrong (since n270 is an Atom processor). Apart from architecture, the
package versions of libvirt-bin are identical: 1.2.1-1~bpo70+1.
/usr/share/libvirt/cpu_map.xml files are identical. Is this a known
issue? Details for one server are:

# cat /proc/cpuinfo| head -n 26
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2650L 0 @ 1.80GHz
stepping        : 7
microcode       : 0x70d
cpu MHz         : 1800.054
cache size      : 20480 KB
physical id     : 0
siblings        : 16
core id         : 0
cpu cores       : 8
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64
monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1
sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat
xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips        : 3600.10
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

...
</proc/cpuinfo for processors 1..31 snipped here for brevity>

Check if all 32 CPUs have *exactly* the same flags, I remember an
issue when we reported a wrong cpu because the probing code was
scheduled on one of them which had one flag missing.  If package and
cpu_map.xml are the same, this is the only thing I can think of.
Simple 'grep ^flags /proc/cpuinfo | sort -u' should do.  If only one
line is printed out than I don't know where the problem might be...

Martin

# Running Wheezy-amd64 libvirt-bin1.2.1-1~bpo70+1
# virsh capabilities

         <cpu>
           <arch>x86_64</arch>
           <model>SandyBridge</model>
           <vendor>Intel</vendor>
           <topology sockets='2' cores='8' threads='2'/>
           <feature name='pdpe1gb'/>
           <feature name='osxsave'/>
           <feature name='dca'/>
           <feature name='pcid'/>
           <feature name='pdcm'/>
           <feature name='xtpr'/>
           <feature name='tm2'/>
           <feature name='est'/>
           <feature name='smx'/>
           <feature name='vmx'/>
           <feature name='ds_cpl'/>
           <feature name='monitor'/>
           <feature name='dtes64'/>
           <feature name='pbe'/>
           <feature name='tm'/>
           <feature name='ht'/>
           <feature name='ss'/>
           <feature name='acpi'/>
           <feature name='ds'/>
           <feature name='vme'/>
         </cpu>


# Running Wheezy-i386 libvirt-bin1.2.1-1~bpo70+1
# virsh capabilities

         <cpu>
           <arch>x86_64</arch>
           <model>n270</model>
           <vendor>Intel</vendor>
           <topology sockets='2' cores='8' threads='2'/>
           <feature name='lahf_lm'/>
           <feature name='lm'/>
           <feature name='rdtscp'/>
           <feature name='pdpe1gb'/>
           <feature name='avx'/>
           <feature name='osxsave'/>
           <feature name='xsave'/>
           <feature name='aes'/>
           <feature name='tsc-deadline'/>
           <feature name='popcnt'/>
           <feature name='x2apic'/>
           <feature name='sse4.2'/>
           <feature name='sse4.1'/>
           <feature name='dca'/>
           <feature name='pcid'/>
           <feature name='pdcm'/>
           <feature name='xtpr'/>
           <feature name='cx16'/>
           <feature name='tm2'/>
           <feature name='est'/>
           <feature name='smx'/>
           <feature name='vmx'/>
           <feature name='ds_cpl'/>
           <feature name='dtes64'/>
           <feature name='pclmuldq'/>
           <feature name='pbe'/>
           <feature name='tm'/>
           <feature name='ht'/>
           <feature name='ss'/>
           <feature name='acpi'/>
           <feature name='ds'/>
           <feature name='pse36'/>
         </cpu>

Kind regards

Struan Bartlett