There have been queries about the OpenStack interface for CAT:

http://bugzilla.redhat.com/show_bug.cgi?id=1299678 Comment 2 says: Sahid Ferdjaoui 2016-01-19 10:58:48 EST A spec will have to be addressed, after a first look this feature needs some work in several components of Nova to maintain/schedule/consume host's cache. I can work on that spec and implementation it when libvirt will provides information about cache and feature to use it for guests. I could add a comment about parameters to resctrltool, but since this depends on the libvirt interface, it would be good to know what the libvirt interface exposes first. I believe it should be essentially similar to OpenStack's "reserved_host_memory_mb": Set the reserved_host_memory_mb to reserve RAM for host processes. For the purposes of testing I am going to use the default of 512 MB: reserved_host_memory_mb=512 But rather use: rdt_cat_cache_reservation=type=code/data/both,size=10mb,cacheid=2; type=code/data/both,size=2mb,cacheid=1;... (per-vcpu). Where cache-id is optional. What is cache-id (from Documentation/x86/intel_rdt_ui.txt on recent kernel sources): Cache IDs --------- On current generation systems there is one L3 cache per socket and L2 caches are generally just shared by the hyperthreads on a core, but this isn't an architectural requirement. We could have multiple separate L3 caches on a socket, multiple cores could share an L2 cache. So instead of using "socket" or "core" to define the set of logical cpus sharing a resource we use a "Cache ID". At a given cache level this will be a unique number across the whole system (but it isn't guaranteed to be a contiguous sequence, there may be gaps). To find the ID for each logical CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id

WHAT THE USER NEEDS TO SPECIFY FOR VIRTUALIZATION (KVM-RT) ========================================================== For virtualization the following scenario is desired, on a given socket: * VM-A with VCPUs VM-A.vcpu-1, VM-A.vcpu-2. * VM-B with VCPUs VM-B.vcpu-1, VM-B.vcpu-2. With one realtime workload on each vcpu-2. Assume VM-A.vcpu-2 on pcpu 3. Assume VM-B.vcpu-2 on pcpu 5. Assume pcpus 0-5 on cacheid 0. We want VM-A.vcpu-2 to have a certain region of cache reserved, and VM-B.vcpu-2 as well. vcpu-1 for both VMs can use the default group (that is not have reserved L3 cache). This translates to the following resctrltool-style reservations: res.vm-a.vcpu-2 type=both,size=VM-A-RESSIZE,cache-id=0 res.vm-b.vcpu-2 type=both,size=VM-B-RESSIZE,cache-id=0 Which translate to the following in resctrlfs: res.vm-a.vcpu-2 type=both,size=VM-A-RESSIZE,cache-id=0 type=both,size=default-size,cache-id=1 ... res.vm-b.vcpu-2 type=both,size=VM-B-RESSIZE,cache-id=0 type=both,size=default-size,cache-id=1 ... Which is what we want, since the VCPUs are pinned. res.vm-a.vcpu-1 and res.vm-b.vcpu-1 don't need to be assigned to any reservation, which means they'll remain on the default group.

RESTRICTIONS TO THE SYNTAX ABOVE ================================ Rules for the parameters: * type=code must be paired with type=data entry.

ABOUT THE LIST INTERFACE ======================== About an interface for listing the reservations of the system to OpenStack. I think that what OpenStack needs is to check, before starting a guest on a given host, that there is sufficient space available for the reservation. To do that, it can: 1) resctrltool list (the end of the output mentions how much free space available there is), or via resctrlfs directly (have to lock the filesystem, read each directory, AND each schemata, and count number of zero bits). 2) Via libvirt Should fix resctrltool/API to list amount of contiguous free space

> > > > This translates to the following resctrltool-style reservations: > > > > res.vm-a.vcpu-2 > > > > type=both,size=VM-A-RESSIZE,cache-id=0 > > > > res.vm-b.vcpu-2 > > > > type=both,size=VM-B-RESSIZE,cache-id=0 > > > > Which translate to the following in resctrlfs: > > > > res.vm-a.vcpu-2 > > > > type=both,size=VM-A-RESSIZE,cache-id=0 > > type=both,size=default-size,cache-id=1 > > ... > > > > res.vm-b.vcpu-2 > > > > type=both,size=VM-B-RESSIZE,cache-id=0 > > type=both,size=default-size,cache-id=1 > > ... > >

> > Which is what we want, since the VCPUs are pinned. > > > > > > res.vm-a.vcpu-1 and res.vm-b.vcpu-1 don't need to > > be assigned to any reservation, which means they'll > > remain on the default group. > > You've showing type=both here which IIUC, means data > and instruction cache. No, type=both is non-cdp hosts (data and instructions reservations shared). type=data,type=code is for cdp hosts (data and instructions reservations separate). > Is that configuring one cache > that serves both purposes ? Yes. > Do we need to be able > to configure them independantly. Yes. > > RESTRICTIONS TO THE SYNTAX ABOVE > > ================================ > > > > Rules for the parameters: > > * type=code must be paired with type=data entry. > > What does this mean exactly when configuring guests ? Do > we have to configure data + instruction cache on the same > cache ID, do they have to be the same size, or are they > completely independant ? This means that a user can't specify this reservation: type=data,size=10mb,cache-id=1 They have to specify _both_ code and data sizes: type=data,size=10mb,cache-id=1; type=code,size=2mb,cache-id=1 Now a single both reservation is valid: type=both,size=10mb,cache-id=1

> > ABOUT THE LIST INTERFACE > > ======================== > > > > About an interface for listing the reservations > > of the system to OpenStack. > > > > I think that what OpenStack needs is to check, before > > starting a guest on a given host, that there is sufficient > > space available for the reservation. > > > > To do that, it can: > > > > 1) resctrltool list (the end of the output mentions > > how much free space available there is), or > > via resctrlfs directly (have to lock the filesystem, > > read each directory, AND each schemata, and count > > number of zero bits). > > 2) Via libvirt > > > > Should fix resctrltool/API to list amount of contiguous free space > > OpenStack, should just use libvirt APIs exclusively - there should not > be any need for it to use other tools if we've designed the libvirt API > correctly. Got it. -- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

On Tue, Jan 10, 2017 at 02:18:41PM -0200, Marcelo Tosatti wrote: > > There have been queries about the OpenStack interface > for CAT: > > http://bugzilla.redhat.com/show_bug.cgi?id=1299678 > > Comment 2 says: > Sahid Ferdjaoui 2016-01-19 10:58:48 EST > A spec will have to be addressed, after a first look this feature needs > some work in several components of Nova to maintain/schedule/consume > host's cache. I can work on that spec and implementation it when libvirt > will provides information about cache and feature to use it for guests. > > I could add a comment about parameters to resctrltool, but since > this depends on the libvirt interface, it would be good to know > what the libvirt interface exposes first. > > I believe it should be essentially similar to OpenStack's > "reserved_host_memory_mb": > > Set the reserved_host_memory_mb to reserve RAM for host > processes. For > the purposes of testing I am going to use the default of 512 MB: > reserved_host_memory_mb=512 > > But rather use: > > rdt_cat_cache_reservation=type=code/data/both,size=10mb,cacheid=2; > type=code/data/both,size=2mb,cacheid=1;... > > (per-vcpu). > > Where cache-id is optional. > > What is cache-id (from Documentation/x86/intel_rdt_ui.txt on recent > kernel sources): > Cache IDs > --------- > On current generation systems there is one L3 cache per socket and L2 > caches are generally just shared by the hyperthreads on a core, but this > isn't an architectural requirement. We could have multiple separate L3 > caches on a socket, multiple cores could share an L2 cache. So instead > of using "socket" or "core" to define the set of logical cpus sharing > a resource we use a "Cache ID". At a given cache level this will be a > unique number across the whole system (but it isn't guaranteed to be a > contiguous sequence, there may be gaps). To find the ID for each > logical > CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id > > > WHAT THE USER NEEDS TO SPECIFY FOR VIRTUALIZATION (KVM-RT) > ========================================================== > > For virtualization the following scenario is desired, > on a given socket: > > * VM-A with VCPUs VM-A.vcpu-1, VM-A.vcpu-2. > * VM-B with VCPUs VM-B.vcpu-1, VM-B.vcpu-2. > > With one realtime workload on each vcpu-2. > > Assume VM-A.vcpu-2 on pcpu 3. > Assume VM-B.vcpu-2 on pcpu 5. > > Assume pcpus 0-5 on cacheid 0. > > We want VM-A.vcpu-2 to have a certain region of cache reserved, > and VM-B.vcpu-2 as well. vcpu-1 for both VMs can use the default group > (that is not have reserved L3 cache). > > This translates to the following resctrltool-style reservations: > > res.vm-a.vcpu-2 > > type=both,size=VM-A-RESSIZE,cache-id=0 > > res.vm-b.vcpu-2 > > type=both,size=VM-B-RESSIZE,cache-id=0 > > Which translate to the following in resctrlfs: > > res.vm-a.vcpu-2 > > type=both,size=VM-A-RESSIZE,cache-id=0 > type=both,size=default-size,cache-id=1 > ... > > res.vm-b.vcpu-2 > > type=both,size=VM-B-RESSIZE,cache-id=0 > type=both,size=default-size,cache-id=1 > ... > > Which is what we want, since the VCPUs are pinned. > > > res.vm-a.vcpu-1 and res.vm-b.vcpu-1 don't need to > be assigned to any reservation, which means they'll > remain on the default group. > > RESTRICTIONS TO THE SYNTAX ABOVE > ================================ > > Rules for the parameters: > * type=code must be paired with type=data entry. > > ABOUT THE LIST INTERFACE > ======================== > > About an interface for listing the reservations > of the system to OpenStack. > > I think that what OpenStack needs is to check, before > starting a guest on a given host, that there is sufficient > space available for the reservation. > > To do that, it can: > > 1) resctrltool list (the end of the output mentions > how much free space available there is), or > via resctrlfs directly (have to lock the filesystem, > read each directory, AND each schemata, and count > number of zero bits). > 2) Via libvirt > > Should fix resctrltool/API to list amount of contiguous free space > BTW. Elements of the libvirt CAT interface: