Re: [libvirt] [PATCHv5 00/19] Introduce x86 Cache Monitoring Technology (CMT)

Seems odd in this example if you think one 'vcpus' is a copy of another. Actually the two 'vcpus' can be assigned with different vcpu setting. Let's introduce some background. resource for allocation group and monitor group. And the number of hardware CLOSID, the hardware class of service ID, determines the number of allocation groups could be created. And the number of hardware RMID, and which is hardware resource monitor ID, determines the monitor groups could be created simultaneously in host. Normally we have more RMIDs and CLOSIDs, that is, we can create more allocations than monitors. Based on this hardware design, kernel introduces the resctrl file system and which has the capability to create more monitors than allocations, and create more than one monitors for each allocation. Allocations can only be created under root directory of '/sys/fs/resctrl' with one exception that the root directory itself is also an allocation, which is called default allocation. For monitors, could be created under default allocation's 'mon_groups' directory, which is directory '/sys/fs/resctrl/mon_groups', or be created under other allocation's 'mon_groups' directory, for example '/sys/fs/resctrl/p0/mon_groups' for allocation 'p0'. Each directory under the allocation's 'mon_group' occupies one RMID, and you can create several directories under this directory. Each allocation itself occupies one RMID and creates a monitor monitoring the cache or memory bandwidth utilization for all CPUs using this RMID. With the help of kernel scheduler, the hardware RMID is assigned to the CPU servicing current Linux thread at time of CPU context switch. That is the RMID could be tracked through PID of vcpu. Here, since both <cachetune> and <monitor> have the same 'vcpus' attribute but it may pointer to different vcpu list. The rules are: 1. In each <cachetune> entry more than one monitors could be specified. 2. In each <cachetune> entry up to one allocation could be specified. 3. The allocation is using the vcpu list specified in <cachetune> attribute 'vcpus'. 4. A monitor has the same vcpu list as allocation is allowed, and this monitor is allocation's default monitor. 5. A monitor has a subset vcpu list of allocation is allowed. 6. For non-default monitors, any vcpu list overlap is not permitted. Since we treat both memorytune and cachetune as the allocation, These rules are applicable to memoryBW allocation if we replace the <cachetune> with <memorytune>. So following XML are all valid: <cputune> <cachetune vcpus='1'> <cache id='0' level='3' type='code' size='7680' unit='KiB'/> <cache id='1' level='3' type='data' size='3840' unit='KiB'/> <monitor level='3' vcpus='1'/> </cachetune> <cachetune vcpus='2-5'> <cache id='0' level='3' type='code' size='7680' unit='KiB'/> <cache id='1' level='3' type='data' size='3840' unit='KiB'/> <monitor level='3' vcpus='2'/> <monitor level='3' vcpus='3,5'/> <monitor level='3' vcpus='4'/> <monitor level='3' vcpus='2-5'/> </cachetune> <cachetune vcpus='6'> <monitor level='3' vcpus='6'/> </cachetune> </cputune> Any of following <cachetune> entry is invalid: <cachetune vcpus='6'> <monitor level='3' vcpus='7'/> </cachetune> <cachetune vcpus='2-5'> <monitor level='3' vcpus='2-4'/> <monitor level='3' vcpus='2'/> </cachetune> <cachetune vcpus='6'> <cache id='0' level='3' type='code' size='7680' unit='KiB'/> <cache id='1' level='3' type='data' size='3840' unit='KiB'/> <monitor level='3' vcpus='7'/> </cachetune> See above explanation. Actually 'cpu.cache.1.name' shows the cache monitor ID, content of @virResctrlMonitor.id. Because 'cpu.cache.x.id' is more confusion for describing the cache since the 'cache ID' describes the cache bank index in some places. I also think 'vcpus_4-6' is more reasonable than '4-6' to describe a monitor's name or id. If you insist '4-6' is better, I could change that. vcpu list "5,7,9" is valid here and the monitor's output would be: cpu.cache.monitor.count=... cpu.cache.0.name=vcpus_5,7,9 cpu.cache.0.vcpus=5,7,9 cpu.cache.0.bank.count=2 cpu.cache.0.bank.0.id=0 cpu.cache.0.bank.0.bytes=4505600 cpu.cache.0.bank.0.id=0 cpu.cache.0.bank.1.bytes=4505600 cpu.cache.0.bank.1.id=0 These numbers in my example are not the real number that I get from a system, they may not match up. .bytes reports the cache utilization information of current monitor, it might be less than the cache allocated from current allocation. It is reasonable that we allocate 10MB cache to vcpus 0 and vcpus 0 only used part of that cache resource. I have some trouble in making CAT work in my test machine, I'll try to catch some real numbers to illustrate these numbers when I fixed the CAT issue. I understand, and noticed you also submit your patch series to community and has a lot of heavy work on task of review. In V5, I added some missing part. Yes you told me about this, my bad. Will be fixed. Thank you very much for your review. Huaqiang