Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking.

<domain type='xen' id='42'>    ...    <features>        <pae/>        <acpi/>        <apic/>        <cpuid>            <mask level='1' register='ebx'>                xxxx:xxxx:0000:1010:xxxx:xxxx:xxxx:xxxx            </mask>            <mask level='1' register='ecx'>                xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xx1x:xxxx            </mask>            <mask level='1' register='edx'>                xxx1:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx            </mask>            <mask level='80000001' register='ecx'>                xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xx1x            </mask>            <mask level='80000008' register='ecx'>                xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xx00:1001            </mask>        </cpuid>    </features>    ... </domain>

Jiri Denemark wrote: > Hi, > > We need to provide support for CPU ID masking. Xen and VMware ESX are examples > of current hypervisors which support such masking. > > My proposal is to define new 'cpuid' feature advertised in guest > capabilities: ... > <domain type='xen' id='42'> > ... > <features> > <pae/> > <acpi/> > <apic/> > <cpuid> > <mask level='1' register='ebx'> > xxxx:xxxx:0000:1010:xxxx:xxxx:xxxx:xxxx > </mask> ... > What are your opinions about this? I think it's too low-level, and the structure is x86-specific. QEMU and KVM compute their CPUID response based on arguments to the -cpu argument, e.g.: -cpu core2duo,model=23,+ssse3,+lahf_lm I think a similar structure makes more sense for libvirt, where the configuration generally avoids big blocks of binary data, and the XML format should suit other architectures as well.

On 09/02/2009 07:09 PM, Daniel P. Berrange wrote: >On Wed, Sep 02, 2009 at 11:59:39AM -0400, Jim Paris wrote: >>Jiri Denemark wrote: >>>Hi, >>> >>>We need to provide support for CPU ID masking. Xen and VMware ESX are >>>examples >>>of current hypervisors which support such masking. >>> >>>My proposal is to define new 'cpuid' feature advertised in guest >>>capabilities: >>... >>><domain type='xen' id='42'> >>> ... >>> <features> >>> <pae/> >>> <acpi/> >>> <apic/> >>> <cpuid> >>> <mask level='1' register='ebx'> >>> xxxx:xxxx:0000:1010:xxxx:xxxx:xxxx:xxxx >>> </mask> >>... >>>What are your opinions about this? >> >>I think it's too low-level, and the structure is x86-specific. QEMU >>and KVM compute their CPUID response based on arguments to the -cpu >>argument, e.g.: >> >> -cpu core2duo,model=23,+ssse3,+lahf_lm >> >>I think a similar structure makes more sense for libvirt, where the >>configuration generally avoids big blocks of binary data, and the >>XML format should suit other architectures as well. > >I'm going back& forth on this too. We essentially have 3 options > > - Named CPU + flags/features > - CPUID masks > - Allow either > > >If we do either of the first two, we have to translate between the >two formats for one or more of the hypervisors. For the last one we >are just punting the problem off to applications. > > >If we choose CPUID, and made QEMU driver convert to named CPU + flags >we'd be stuck for non-x86 as you say. Why is that? cpu model + flags may apply for other arch too.

>If we chose named CPU + flags, and made VMWare/Xen convert to raw >CPUID we'd potentially loose information if user had defined a config >with a raw CPUID mask outside context of libvirt. > >The other thing to remember is that CPUID also encodes sockets/cores/ >threads topology data, and it'd be very desirable to expose that in >a sensible fashion (ie not a bitmask). > >On balance i'm currently leaning to named CPU + flags + expliciti >topology data because although its harder to implement for Xen/VMWare >I think its much nicer to applications& users. We might loose a tiny >bit of data in the CPU -> named/flags conversion for Xen/VMWare but >I reckon we can get it good enough that most people won't really care >about that. > >Daniel There are 2 more issues to consider: 1. The VMW approach with all the cpuid bits might be ok, the problem is to map it into qemu model, will libvirt to that?

2. If we use the qemu approach, the host information (cpuids) need to travel to higher mgmt level in order to allow computation of greatest common denominator.

Speaking from an x86 angle,providing an ability to enable or disable high level constructs like SSE instead of low level constructs, will make it easy to understand.

Thanks Mukesh On Thu, Sep 3, 2009 at 3:09 PM, Daniel P. Berrange&lt;berrange(a)redhat.com&gt; wrote: > On Thu, Sep 03, 2009 at 11:19:47AM +0300, Dor Laor wrote: >> On 09/02/2009 07:09 PM, Daniel P. Berrange wrote: >>> On Wed, Sep 02, 2009 at 11:59:39AM -0400, Jim Paris wrote: >>>> Jiri Denemark wrote: >>>>> Hi, >>>>> >>>>> We need to provide support for CPU ID masking. Xen and VMware ESX are >>>>> examples >>>>> of current hypervisors which support such masking. >>>>> >>>>> My proposal is to define new 'cpuid' feature advertised in guest >>>>> capabilities: >>>> ... >>>>> <domain type='xen' id='42'> >>>>> ... >>>>> <features> >>>>> <pae/> >>>>> <acpi/> >>>>> <apic/> >>>>> <cpuid> >>>>> <mask level='1' register='ebx'> >>>>> xxxx:xxxx:0000:1010:xxxx:xxxx:xxxx:xxxx >>>>> </mask> >>>> ... >>>>> What are your opinions about this? >>>> >>>> I think it's too low-level, and the structure is x86-specific. QEMU >>>> and KVM compute their CPUID response based on arguments to the -cpu >>>> argument, e.g.: >>>> >>>> -cpu core2duo,model=23,+ssse3,+lahf_lm >>>> >>>> I think a similar structure makes more sense for libvirt, where the >>>> configuration generally avoids big blocks of binary data, and the >>>> XML format should suit other architectures as well. >>> >>> I'm going back& forth on this too. We essentially have 3 options >>> >>> - Named CPU + flags/features >>> - CPUID masks >>> - Allow either >>> >>> >>> If we do either of the first two, we have to translate between the >>> two formats for one or more of the hypervisors. For the last one we >>> are just punting the problem off to applications. >>> >>> >>> If we choose CPUID, and made QEMU driver convert to named CPU + flags >>> we'd be stuck for non-x86 as you say. >> >> Why is that? cpu model + flags may apply for other arch too. > > If we have CPUID in the XML, there is no meaningful CPUID register > representation for sparc/ppc/arm/etc. It is an x86 concept, which > is almost certainly why QEMU uses named CPU models + named flags > instead of CPUID as is public facing config. > > Xen/VMWare of course don't have this limitation since they only > really care about x86. > > So really QEMU's CPU model + flags approach is more generic albeit > being much more verbose to achieve the same level of expressivity. > >>> If we chose named CPU + flags, and made VMWare/Xen convert to raw >>> CPUID we'd potentially loose information if user had defined a config >>> with a raw CPUID mask outside context of libvirt. >>> >>> The other thing to remember is that CPUID also encodes sockets/cores/ >>> threads topology data, and it'd be very desirable to expose that in >>> a sensible fashion (ie not a bitmask). >>> >>> On balance i'm currently leaning to named CPU + flags + expliciti >>> topology data because although its harder to implement for Xen/VMWare >>> I think its much nicer to applications& users. We might loose a tiny >>> bit of data in the CPU -> named/flags conversion for Xen/VMWare but >>> I reckon we can get it good enough that most people won't really care >>> about that. >>> >>> Daniel >> >> There are 2 more issues to consider: >> 1. The VMW approach with all the cpuid bits might be ok, the problem is >> to map it into qemu model, will libvirt to that? > > THe problem is that CPUID is not viable for non-x86 archs so can't > really be used as our master representation > >> 2. If we use the qemu approach, the host information (cpuids) need to >> travel to higher mgmt level in order to allow computation of >> greatest common denominator. > > Yes, whatever we decide for exposing guest CPU model/flags/etc should > be equally applied to the libvirt capabilities XML so that apps can > query physical host data > > Regards, > Daniel > -- > |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| > |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| > |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| > > -- > Libvir-list mailing list > Libvir-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/libvir-list >

What about another approach for the cpuid issue: I think that dealing with specific flags is pretty error prone on all levels - virt-mgr, libvirt, qemu, migration, and even the guest.

We can't just 'invent' new cpu modules that will be a combination of flags we have on the host. It might work for some kernels/apps but it might break others. In addition to cpuid we have the stepping and more MSRs that needed to be hidden/exposed.

The offer: Libvirt, virt-mgr, qemu will not deal with lowlevel bits of the cpuid. We'll use predefined cpu modules to be emulated. These predefined modules will represent a real module that was once shipped by Intel/AMD. We'll write an additional tool, not under libvirt, that will be able to calculate all the virtual cpus that can run over the physical cpu. This tool will be the one messing with /proc/cpuinfo, etc. Example (theoretical): Physical cpu is "Intel(R) Core(TM)2 Duo CPU T9600 @ 2.80GHz" the output of the tool will be: shell>computeVirtCpuCapabilities core2duo core solo 486 pentium3 .. Libvirt will only expose the real physical cpu and all of the outputs above. Higher level mgmt will compute the best -cpu to pick for the VM, and it will take account the user needs for performance or for migration flexibility. The cons: - Simple for all levels - Fast implementation - Accurate representative of real cpus The pros: - none - we should write the tool anyway - Maybe we hide some of the possibilities, but as said above, it is safer and friendlier.

I also recommend of adding a field to be added next to the cpu for flexibility and possible future changes + dealing with problematic bios with NX bit disabled. Regards, Dor > > Thanks > > Mukesh > > On Thu, Sep 3, 2009 at 3:09 PM, Daniel P. > Berrange&lt;berrange(a)redhat.com&gt; wrote: >> On Thu, Sep 03, 2009 at 11:19:47AM +0300, Dor Laor wrote: >>> On 09/02/2009 07:09 PM, Daniel P. Berrange wrote: >>>> On Wed, Sep 02, 2009 at 11:59:39AM -0400, Jim Paris wrote: >>>>> Jiri Denemark wrote: >>>>>> Hi, >>>>>> >>>>>> We need to provide support for CPU ID masking. Xen and VMware ESX >>>>>> are >>>>>> examples >>>>>> of current hypervisors which support such masking. >>>>>> >>>>>> My proposal is to define new 'cpuid' feature advertised in guest >>>>>> capabilities: >>>>> ... >>>>>> <domain type='xen' id='42'> >>>>>> ... >>>>>> <features> >>>>>> <pae/> >>>>>> <acpi/> >>>>>> <apic/> >>>>>> <cpuid> >>>>>> <mask level='1' register='ebx'> >>>>>> xxxx:xxxx:0000:1010:xxxx:xxxx:xxxx:xxxx >>>>>> </mask> >>>>> ... >>>>>> What are your opinions about this? >>>>> >>>>> I think it's too low-level, and the structure is x86-specific. QEMU >>>>> and KVM compute their CPUID response based on arguments to the -cpu >>>>> argument, e.g.: >>>>> >>>>> -cpu core2duo,model=23,+ssse3,+lahf_lm >>>>> >>>>> I think a similar structure makes more sense for libvirt, where the >>>>> configuration generally avoids big blocks of binary data, and the >>>>> XML format should suit other architectures as well. >>>> >>>> I'm going back& forth on this too. We essentially have 3 options >>>> >>>> - Named CPU + flags/features >>>> - CPUID masks >>>> - Allow either >>>> >>>> >>>> If we do either of the first two, we have to translate between the >>>> two formats for one or more of the hypervisors. For the last one we >>>> are just punting the problem off to applications. >>>> >>>> >>>> If we choose CPUID, and made QEMU driver convert to named CPU + flags >>>> we'd be stuck for non-x86 as you say. >>> >>> Why is that? cpu model + flags may apply for other arch too. >> >> If we have CPUID in the XML, there is no meaningful CPUID register >> representation for sparc/ppc/arm/etc. It is an x86 concept, which >> is almost certainly why QEMU uses named CPU models + named flags >> instead of CPUID as is public facing config. >> >> Xen/VMWare of course don't have this limitation since they only >> really care about x86. >> >> So really QEMU's CPU model + flags approach is more generic albeit >> being much more verbose to achieve the same level of expressivity. >> >>>> If we chose named CPU + flags, and made VMWare/Xen convert to raw >>>> CPUID we'd potentially loose information if user had defined a config >>>> with a raw CPUID mask outside context of libvirt. >>>> >>>> The other thing to remember is that CPUID also encodes sockets/cores/ >>>> threads topology data, and it'd be very desirable to expose that in >>>> a sensible fashion (ie not a bitmask). >>>> >>>> On balance i'm currently leaning to named CPU + flags + expliciti >>>> topology data because although its harder to implement for Xen/VMWare >>>> I think its much nicer to applications& users. We might loose a >>>> tiny >>>> bit of data in the CPU -> named/flags conversion for Xen/VMWare but >>>> I reckon we can get it good enough that most people won't really care >>>> about that. >>>> >>>> Daniel >>> >>> There are 2 more issues to consider: >>> 1. The VMW approach with all the cpuid bits might be ok, the problem is >>> to map it into qemu model, will libvirt to that? >> >> THe problem is that CPUID is not viable for non-x86 archs so can't >> really be used as our master representation >> >>> 2. If we use the qemu approach, the host information (cpuids) need to >>> travel to higher mgmt level in order to allow computation of >>> greatest common denominator. >> >> Yes, whatever we decide for exposing guest CPU model/flags/etc should >> be equally applied to the libvirt capabilities XML so that apps can >> query physical host data >> >> Regards, >> Daniel >> -- >> |: Red Hat, Engineering, London -o- >> http://people.redhat.com/berrange/ :| >> |: http://libvirt.org -o- http://virt-manager.org -o- >> http://ovirt.org :| >> |: http://autobuild.org -o- >> http://search.cpan.org/~danberr/ :| >> |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B >> 9505 :| >> >> -- >> Libvir-list mailing list >> Libvir-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/libvir-list >>

On Tue, Oct 13, 2009 at 02:56:41AM -0400, john cooper wrote: > Dor Laor wrote: >> What about another approach for the cpuid issue: >> I think that dealing with specific flags is pretty error prone on all >> levels - virt-mgr, libvirt, qemu, migration, and even the guest. > > ..and performance verification, QA, and the average end user. > Unless we reduce all possible combinations of knob settings > into a few well thought out lumped models the complexity can > be overwhelming. That is a policy decision for applications to make. libvirt should expose the fine grained named CPU models + arbitrary flags, and other bits of info as appropriate (eg formal model for core/socket topology). Apps can decide whether they want to turn that into a higher level concept where admins pick one of a handful of common setups, or expose the full level of control

Daniel