hi all


Sorry for resend this again, I forget to add subject :( ... 


Thanks all for the comment for previous RFC version, I summary all input here also with Opens which I am not sure, can you please review again?

 

#Propose Changes

 

## Expose cache information into capabilities

 

 

Extend capabilities to expose all level cache resource information and should tell the topology of caches also tell which kinds of resources can be tuned/allocation

 

There information comes from /sys/devices/system/cpu/cpu0/cache/index${level}/size also /sys/fs/resctrl/ (this is CAT sys interface in linux kernel)

 

virsh capabilities 

 

    <cache>

      <bank id='0, 'type="l3" size="56320" units="KiB" cpus="0,1,2,6,7,8"/>  <--------------------- level 3 cache is per socket, so group them by socket id

           <control unit="KiB" min="2816"/>

      <bank id='1', type="l3" size="56320" units="KiB" cpus="3,4,5,9,10,11"/>

      <bank id='2' type="l2" size="256" units="KiB" cpus="0"/>

      <bank id='3' type="l2" size="256" units="KiB" cpus="1"/>

      <bank id='4' type="l2" size="256" units="KiB" cpus="2"/>

      <bank id='5' type="l2" size="256" units="KiB" cpus="3"/>

      <bank id='6' type="l2" size="256" units="KiB" cpus="4"/>

...

     <cache> 

 

Opens

1.      how about add socket id to bank for bank type = l3 ? 

2.      do we really want to expose l2/l3 cache for now , they are per core resource and linux kernel don't support l2 yet (depend no hardware)?

3.      if enable CDP in resctrl, for bank type=l3 , it will be split to l3data l3code, should expose this ability.

      <bank type="l3" size="56320" units="KiB" cpus="0,1,2,6,7,8"/>  <--------------------- level 3 cache is per socket, so group them by socket id

           <control unit="KiB" min="2816" cdp="enabled"/>

 

## Provide a new API to get the avail cache on each bank, such as the output are:

 

   id=0

   type=l3

   avail=56320

   total = ?? <--------- do we need this?

 

   id=1

   type=l3

   avail=56320

 

   id=3

   type=l2

   avail=256

 

Opens:

·         Don't expose the avail cache information if the host can not do the allocation of that type cache(eg, for l2 currently) ?

·         We can not make all of the cache , the reservation amount is the min_cbm_len (=1) * min_unit . 

·         do we need to expose total? 

 

 

 

##  enable CAT for a domain

 

1 Domain XML changes

 

   <cputune>

       <cache id="1" host_id="0" type="l3" size="5632" unit="KiB"/>

       <cache id="2" host_id="1" type="l3" size="5632" unit="KiB"/> 

 

       <cpu_cache vcpus="0-3" id="1"/>

       <cpu_cache vcpus="4-7" id="2"/>

       <iothread_cache iothreads="0-1" id="1"/>

       <emulator_cache id="2"/>

   </cputune>

 

2. Extend cputune command ?

 

 

Opens:

 

1. Do we accept to extend existed API ? or using new API/virsh?

2. How to calculate cache size -> CBM bit? 

 

eg: 

5632/ 2816 = 2 bits

5733/ 2816 = 2 bits or 3 bits?

 

 

 

## Restriction for using cache tune on multiple sockets' host.

 

The l3 cache is per socket resource, kernel need to know about what's affinity looks like, so for a VM which running on a multiple socket's host, it should have NUMA setting or vcpuset pin setting. Or cache tune will fail.

 

[1] kernel support https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/tree/arch/x86/kernel/cpu/intel_rdt.c?h=x86/cache

 

[2] libvirt PoC(not finished yet) https://github.com/taget/libvirt/commits/cat_new


--
Best regards 
- Eli

天涯无处不重逢
a leaf duckweed belongs to the sea , where not to meet in life