Re: [libvirt-users] 'virsh capabilities' on Debian Wheezy-amd64 reports different cpu to Wheezy-i386 (on same hardware)

On 03/03/2014 13:42, Martin Kletzander wrote: > On Mon, Mar 03, 2014 at 11:15:51AM +0000, Struan Bartlett wrote: >> On 03/03/2014 10:55, Martin Kletzander wrote: >>> On Mon, Mar 03, 2014 at 10:47:03AM +0000, Struan Bartlett wrote: >>>> On 03/03/2014 10:44, Martin Kletzander wrote: >>>>> On Mon, Mar 03, 2014 at 10:30:11AM +0000, Struan Bartlett wrote: >>>>>> Hi Martin >>>>>> >>>>>> Thanks for your response. Here's the output of that grep: >>>>>> >>>>>> # grep ^flags /proc/cpuinfo | sort -u >>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca >>>>>> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall >>>>>> nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl >>>>>> xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor >>>>>> ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 >>>>>> x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat xsaveopt >>>>>> pln pts dtherm tpr_shadow vnmi flexpriority ept vpid >>>>>> >>>>>> I've just managed to install the libvirt-bin:amd64 package on the same >>>>>> machine, on the wheezy-i386 distribution. The output of 'virsh >>>>>> capabilities' is now reported correctly, and my VMs that required >>>>>> SandyBridge are now booting. >>>>>> >>>>>> Have you any further suggestions? >>>>>> >>>>> Oh, I missed the fact that it was 32 bit distro. In that case, I >>>>> guess some cpu features won't be available. No other ideas. >>>> Ok thanks Martin. If the amd64 and i386 versions of the package are >>>> behaving differently in this respect, on otherwise the same hardware, >>>> kernel and distro, then I can only presume this is a bug, so my next >>>> step would be to raise this on the development list. >>> Just a guess, but our CPUID instruction might be translated to >>> assembly code differently. You can try running 'cpuid -ir' (or 'cpuid >>> -ir1' if it's the same on all cpus) on both machines, that will give >>> you the results libvirt is processing. If that output is different >>> than there is nothing libvirt can do. If it's the same, it might be a >>> libvirt bug. >> You might be onto something here. The following are run on the same >> physical machine: >> >> # apt-get install cpuid:i386 >> # cpuid -ir1 >/tmp/cpuid-ir1-i386 >> # apt-get install cpuid:amd64 >> # cpuid -ir1 >/tmp/cpuid-ir1-amd64 >> # diff -ubw /tmp/cpuid-ir1-i386 /tmp/cpuid-ir1-amd64 >> --- /tmp/cpuid-ir1-i386 2014-03-03 11:13:00.839208222 +0000 >> +++ /tmp/cpuid-ir1-amd64 2014-03-03 11:13:13.283307377 +0000 >> @@ -10,11 +10,11 @@ >> 00000008 00000000 00000000 00000000 00000000 >> 00000009 00000000 00000000 00000000 00000000 >> 0000000a 07300403 00000000 00000000 00000603 >> -0000000b 00000000 00000000 000000e8 0000002c >> +0000000b 00000000 00000000 0000006f 0000002c > This one has *probably* no significance in libvirt code since we only > call cpuid functions 0x00000001, 0x00000007 and 0x80000001 (which is > enough for us to determine all we need). > >> 0000000c 00000000 00000000 00000000 00000000 >> 0000000d 00000000 00000000 00000000 00000000 >> 80000000 80000008 00000000 00000000 00000000 >> -80000001 00000000 00000000 00000001 2c100000 >> +80000001 00000000 00000000 00000001 2c100800 > But this is something. > > This means in second run the CPU reports it has "syscall" and in the > first one it doesn't. Since "syscall" is x86_64 instruction used > instead of i386's "int 0x80" (IIRC) it makes sense that it is missing > on 32-bit CPU (when the code is running in 32-bit mode). > >> 80000002 20202020 6e492020 286c6574 58202952 >> 80000003 286e6f65 43202952 45205550 36322d35 >> 80000004 204c3035 20402030 30382e31 007a4847 >> >> Could this be enough difference to throw cpu-detection off? >> > Even one bit is enough if that bit represents a flag which is not in > the CPU you are looking for (i.e. SandyBridge). > > I must admin I was too lazy to compare the differences between the > processor definitions you've sent (basically because I'm fairly > certain we have no such mistakes in our code O:-) ). > > But I did that now and indeed, you can see that the difference is only > in the "syscall" feature. You can do that yourselves using the info > from "cpu_map.xml" if you don't believe me ;) > > Martin Martin Many thanks for helping me get to the bottom of this, though could you clarify what you mean when you say that the difference is only in the "syscall" feature? I've looked at cpu_map.xml, and compared the features for n270 (pentiumpro+coreduo+ssse3) with SandyBridge (pentiumpro+Conroe+Penryn+Nehalem+Westmere+pclmuldq+SandyBridge) and am seeing quite a few differences (see below). The main issue though for

me, is that the libvirt architecture need not be representative of the virtualisation architecture. Apart from the issue of CPU detection, a user should otherwise be fine running libvirt-bin:i386 with qemu-system-x86_64:amd64 (which is our aim, leveraging Debian Multiarch

to run only the 64bit binaries that really need 64bit capability or large RAM, while keeping the memory footprint of other binaries small). Is there a workaround? Just customise the cpu_map.xml file?

# diff -ubw /tmp/n270-features /tmp/SandyBridge-features --- /tmp/n270-features 2014-03-03 14:05:02.003880511 +0000 +++ /tmp/SandyBridge-features 2014-03-03 14:04:56.063834563 +0000 @@ -1,25 +1,38 @@ + <feature name='aes'/> <feature name='apic'/> + <feature name='avx'/> <feature name='clflush'/> <feature name='cmov'/> + <feature name='cx16'/> <feature name='cx8'/> <feature name='de'/> <feature name='fpu'/> <feature name='fxsr'/> + <feature name='lahf_lm'/> + <feature name='lm'/> <feature name='mca'/> <feature name='mce'/> <feature name='mmx'/> - <feature name='monitor'/> <feature name='msr'/> <feature name='mtrr'/> <feature name='nx'/> <feature name='pae'/> <feature name='pat'/> + <feature name='pclmuldq'/> <feature name='pge'/> <feature name='pni'/> + <feature name='popcnt'/> <feature name='pse'/> + <feature name='pse36'/> + <feature name='rdtscp'/> <feature name='sep'/> <feature name='sse'/> <feature name='sse2'/> + <feature name='sse4.1'/> + <feature name='sse4.2'/> <feature name='ssse3'/> + <feature name='syscall'/> <feature name='tsc'/> - <feature name='vme'/> + <feature name='tsc-deadline'/> + <feature name='x2apic'/> + <feature name='xsave'/> > >> Struan >> >>>>> Martin >>>>> >>>>>> Struan >>>>>> >>>>>> On 03/03/2014 10:00, Martin Kletzander wrote: >>>>>>> On Fri, Feb 28, 2014 at 03:45:01PM +0000, Struan Bartlett wrote: >>>>>>>> Hi >>>>>>>> >>>>>>>> On a range of Dell servers containing Intel 64bit processors, 'virsh >>>>>>>> capabilities' reports the cpu differently on Debian Wheezy-amd64 and >>>>>>>> Wheezy-i386. The results given by the Wheezy-i386 version seem very >>>>>>>> wrong (since n270 is an Atom processor). Apart from architecture, the >>>>>>>> package versions of libvirt-bin are identical: 1.2.1-1~bpo70+1. >>>>>>>> /usr/share/libvirt/cpu_map.xml files are identical. Is this a known >>>>>>>> issue? Details for one server are: >>>>>>>> >>>>>>>> # cat /proc/cpuinfo| head -n 26 >>>>>>>> processor : 0 >>>>>>>> vendor_id : GenuineIntel >>>>>>>> cpu family : 6 >>>>>>>> model : 45 >>>>>>>> model name : Intel(R) Xeon(R) CPU E5-2650L 0 @ 1.80GHz >>>>>>>> stepping : 7 >>>>>>>> microcode : 0x70d >>>>>>>> cpu MHz : 1800.054 >>>>>>>> cache size : 20480 KB >>>>>>>> physical id : 0 >>>>>>>> siblings : 16 >>>>>>>> core id : 0 >>>>>>>> cpu cores : 8 >>>>>>>> apicid : 0 >>>>>>>> initial apicid : 0 >>>>>>>> fpu : yes >>>>>>>> fpu_exception : yes >>>>>>>> cpuid level : 13 >>>>>>>> wp : yes >>>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge >>>>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe >>>>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good >>>>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 >>>>>>>> monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 >>>>>>>> sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat >>>>>>>> xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid >>>>>>>> bogomips : 3600.10 >>>>>>>> clflush size : 64 >>>>>>>> cache_alignment : 64 >>>>>>>> address sizes : 46 bits physical, 48 bits virtual >>>>>>>> power management: >>>>>>>> >>>>>>>> ... >>>>>>>> </proc/cpuinfo for processors 1..31 snipped here for brevity> >>>>>>>> >>>>>>> Check if all 32 CPUs have *exactly* the same flags, I remember an >>>>>>> issue when we reported a wrong cpu because the probing code was >>>>>>> scheduled on one of them which had one flag missing. If package and >>>>>>> cpu_map.xml are the same, this is the only thing I can think of. >>>>>>> Simple 'grep ^flags /proc/cpuinfo | sort -u' should do. If only one >>>>>>> line is printed out than I don't know where the problem might be... >>>>>>> >>>>>>> Martin >>>>>>> >>>>>>>> # Running Wheezy-amd64 libvirt-bin1.2.1-1~bpo70+1 >>>>>>>> # virsh capabilities >>>>>>>> >>>>>>>> <cpu> >>>>>>>> <arch>x86_64</arch> >>>>>>>> <model>SandyBridge</model> >>>>>>>> <vendor>Intel</vendor> >>>>>>>> <topology sockets='2' cores='8' threads='2'/> >>>>>>>> <feature name='pdpe1gb'/> >>>>>>>> <feature name='osxsave'/> >>>>>>>> <feature name='dca'/> >>>>>>>> <feature name='pcid'/> >>>>>>>> <feature name='pdcm'/> >>>>>>>> <feature name='xtpr'/> >>>>>>>> <feature name='tm2'/> >>>>>>>> <feature name='est'/> >>>>>>>> <feature name='smx'/> >>>>>>>> <feature name='vmx'/> >>>>>>>> <feature name='ds_cpl'/> >>>>>>>> <feature name='monitor'/> >>>>>>>> <feature name='dtes64'/> >>>>>>>> <feature name='pbe'/> >>>>>>>> <feature name='tm'/> >>>>>>>> <feature name='ht'/> >>>>>>>> <feature name='ss'/> >>>>>>>> <feature name='acpi'/> >>>>>>>> <feature name='ds'/> >>>>>>>> <feature name='vme'/> >>>>>>>> </cpu> >>>>>>>> >>>>>>>> >>>>>>>> # Running Wheezy-i386 libvirt-bin1.2.1-1~bpo70+1 >>>>>>>> # virsh capabilities >>>>>>>> >>>>>>>> <cpu> >>>>>>>> <arch>x86_64</arch> >>>>>>>> <model>n270</model> >>>>>>>> <vendor>Intel</vendor> >>>>>>>> <topology sockets='2' cores='8' threads='2'/> >>>>>>>> <feature name='lahf_lm'/> >>>>>>>> <feature name='lm'/> >>>>>>>> <feature name='rdtscp'/> >>>>>>>> <feature name='pdpe1gb'/> >>>>>>>> <feature name='avx'/> >>>>>>>> <feature name='osxsave'/> >>>>>>>> <feature name='xsave'/> >>>>>>>> <feature name='aes'/> >>>>>>>> <feature name='tsc-deadline'/> >>>>>>>> <feature name='popcnt'/> >>>>>>>> <feature name='x2apic'/> >>>>>>>> <feature name='sse4.2'/> >>>>>>>> <feature name='sse4.1'/> >>>>>>>> <feature name='dca'/> >>>>>>>> <feature name='pcid'/> >>>>>>>> <feature name='pdcm'/> >>>>>>>> <feature name='xtpr'/> >>>>>>>> <feature name='cx16'/> >>>>>>>> <feature name='tm2'/> >>>>>>>> <feature name='est'/> >>>>>>>> <feature name='smx'/> >>>>>>>> <feature name='vmx'/> >>>>>>>> <feature name='ds_cpl'/> >>>>>>>> <feature name='dtes64'/> >>>>>>>> <feature name='pclmuldq'/> >>>>>>>> <feature name='pbe'/> >>>>>>>> <feature name='tm'/> >>>>>>>> <feature name='ht'/> >>>>>>>> <feature name='ss'/> >>>>>>>> <feature name='acpi'/> >>>>>>>> <feature name='ds'/> >>>>>>>> <feature name='pse36'/> >>>>>>>> </cpu> >>>>>>>> >>>>>>>> Kind regards >>>>>>>> >>>>>>>> Struan Bartlett >>>>>>>> >>>>>>>>