Re: [libvirt] [RFC PATCH 0/3] RFC for X86 RDT Cache Monitoring Technology (CMT) support

Hi Martin, Thanks for your comments, please see my update inline below. I'll pay attention to these long lines. Thanks for advices. Due to kernel interface removal of the perf events 'cmt,mbmt,mbml', the libvirt will no longer work with latest kernel. Please examine following link for details. https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/c..., This serials is trying to provide the similar functions of this missing part for reporting cmt, mbmt and mbml information. First we only focus on cmt. Comparing with 'CMT perf event already in libvirt', I am trying to implement almost the same output as 'perf.cmt' in the output message of 'domstats', but with another name , such as 'resctrl.cmt' or 'rdt.cmt' (or some others). Another difference is that the underlying implementation is done through the kernel resctrl fs. This serials also attempts to provide a command interface for enabling and disabling cmt feature in scope of whole domain as original perf event based cmt could be controlled, enabled or disabled, through specifying '--enable cmt' or '--disable cmt' while invoking command 'virsh perf <domain>'. Our version is like 'virsh resctrl <domain> --enable' with a difference of no suffix of 'cmt'. The 'cmt' is omitted because the CMT and MBM function are both enabled whenever a valid resctrl fs sub-folder created, there is no way to disable one while enable another one, such as enabling CMT while disabling MBML at the same time. This serials is trying to stick to interfaces exposed by perf event based CMT/MBM and provide an interface substitution for perf event based CMB/MBM, such as the perf based CMT only provides the cache occupancy information for whole domain only. We are also in thinking providing the capability to provide the cache occupancy information based on vcpus groups which may be specified in XML file. For example, if we have following configuration: <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='3-4'/> <vcpupin vcpu='2' cpuset='4-5'/> <vcpupin vcpu='3' cpuset='6-7'/> <cachetune vcpus='0'> <cache id='0' level='3' type='both' size='2816' unit='KiB'/> <cache id='1' level='3' type='both' size='2816' unit='KiB'/> </cachetune> <cachetune vcpus='1-2'> <cache id='0' level='3' type='both' size='2816' unit='KiB'/> <cache id='1' level='3' type='both' size='2816' unit='KiB'/> </cachetune> <rdt-monitoring vcpu='0' enable='yes'> <rdt-monitoring vcpu='1-2' enable='yes'> <rdt-monitoring vcpu='3' enable='yes'> </cputune> The 'domstats' will output following information regarding cmt [root@dl-c200 libvirt]# virsh domstats vm1 --resctrl Domain: 'vm1' rdt.cmt.total=645562 rdt.cmt.vcpu0=104331 rdt.cmt.vcpu1_2=203200 rdt.cmt.vcpu3=340129 Those updates address your comment for " Can you elaborate on how is this different to the CMT perf event that is already in libvirt and can be monitored through domstats API?", any input is welcome. Agree. 'resctrl' do make a lot of confusion to end users. Since the underlying kernel interface combines CAT and MBM features together, what I mean is , the files ' llc_occupancy', ' mbm_local_bytes' and ' mbm_total_bytes' that represent the information of cache, local memory bandwidth, and total memory bandwidth respectively are created automatically and simultaneously for each resctrl group, there is no way to enable one and disable another one. So for a command which affects both cache and memory bandwidth, I would like to use the word 'rdt' as the key command word. Both cache monitoring(CMT) and memory bandwidth monitoring(MBM) are belong to the scope of RDT monitoring. So to replace the confusing word 'resctrl', I'd like to use 'rdtmon' as command name, the command 'virsh resctrl <domain>' would be changed to 'virsh rdtmon <domain>'. Also, here welcoming any suggestions from community. In the POC code of the first version there is no XML changes, and could not be enabled/disabled through XML file. Let's have a discuss and add this function, how about this configuration <cputune> <cachetune vcpus='1-2'> <cache id='0' level='3' type='both' size='2816' unit='KiB'/> <cache id='1' level='3' type='both' size='2816' unit='KiB'/> </cachetune> <rdt-monitoring vcpu='0' enable='no'> <rdt-monitoring vcpu='1-2' enable='yes'> <rdt-monitoring vcpu='3' enable='yes'> </cputune> With upper setting, - Two rdt monitoring groups will be created along with the launch of vm. - <rdt-monitoring vcpu='1-2' enable='yes'> is created automatically due to the setting of <cachetune>. Under resctrl fs, the resctrl allocation and rdt monitoring group are presented in the way of sub-folders, we cannot create two sub-folders under resctrl fs folders for one process. so a resctrl allocation will create a rdt monitoring group as well. This rdt monitoring group could not be disabled in runtime because there is no way to disable resctrl allocation (CAT) in runtime. - <rdt-monitoring vcpu='3' enable='yes'> creates another default enabled rdt monitoring group, and task id (pid associated with vcpu3) will be put into the 'tasks' file. This rdt monitoring over vcpu 3 could be enabled or disabled in runtime through command such as 'virsh rdtmon --enable vcpu3' . The MBM feature will also be enabled or disabled with this command. - <rdt-monitoring vcpu='0' enable='no'> specifies the default CAT state for vcpu0 of domain, which is disabled after launch, and could be changed in runtime. At the time I raised this RFC, there is no design for reporting rdt monitoring information in granularity of cachetune, only report cache /memory bandwidth information for whole domain. But now I'd like to discuss the design that I list above, reporting rdt monitoring Information based on the setting of rdt-monitoring(cachetune) groups. Need your comments.