10:18 a.m.
Dear list,
QEMU gained support for configuring HMAT recently (see
v4.2.0-415-g9b12dfa03a
and friends). HMAT stands for Heterogeneous Memory Attribute Table and
defines
various attributes to NUMA. Guest OS/app can read these information and fine
tune optimization. See [1] for more info (esp. links in the transcript).
QEMU defines so called initiator, which is an attribute to a NUMA node
and if
specified points to another node that has the best performance to this node.
For instance:
-machine hmat=on \
-m 2G,slots=2,maxmem=4G \
-object memory-backend-ram,size=1G,id=m0 \
-object memory-backend-ram,size=1G,id=m1 \
-numa node,nodeid=0,memdev=m0 \
-numa node,nodeid=1,memdev=m1,initiator=0 \
-smp 2,sockets=2,maxcpus=2 \
-numa cpu,node-id=0,socket-id=0 \
-numa cpu,node-id=0,socket-id=1
creates a machine with 2 NUMA nodes, node 0 has CPUs and node 1 has
memory only
and it's initiator is node 0 (yes, HMAT allows you to create CPU-less
"NUMA"
nodes). The initiator of node 0 is not specified, but since the node has at
least one CPU it is initiator to itself (and has to be per specs).
This could be represented by an attribute to our /domain/cpu/numa/cell
element.
For instance like this:
<domain>
<vcpu>2</vcpu>
<cpu>
<numa>
<cell id='0' cpus='0,1' memory='1'
unit='GiB'/>
<cell id='1' memory='1' unit='GiB'
initiator='0'/>
</numa>
</cpu>
</domain>
Then, QEMU allows us to control two other important memory attributes:
1) hmat-lb for Latency and Bandwidth
2) hmat-cache for cache attributes
For example:
-machine hmat=on \
-m 2G,slots=2,maxmem=4G \
-object memory-backend-ram,size=1G,id=m0 \
-object memory-backend-ram,size=1G,id=m1 \
-smp 2,sockets=2,maxcpus=2 \
-numa node,nodeid=0,memdev=m0 \
-numa node,nodeid=1,memdev=m1,initiator=0 \
-numa cpu,node-id=0,socket-id=0 \
-numa cpu,node-id=0,socket-id=1 \
-numa
hmat-lb,initiator=0,target=0,hierarchy=memory,data-type=access-latency,latency=5
\
-numa
hmat-lb,initiator=0,target=0,hierarchy=memory,data-type=access-bandwidth,bandwidth=200M
\
-numa
hmat-lb,initiator=0,target=1,hierarchy=memory,data-type=access-latency,latency=10
\
-numa
hmat-lb,initiator=0,target=1,hierarchy=memory,data-type=access-bandwidth,bandwidth=100M
\
-numa
hmat-cache,node-id=0,size=10K,level=1,associativity=direct,policy=write-back,line=8
\
-numa
hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8
This extends previous example by defining some latencies and cache
attributes.
The node 0 has access latency of 5 ns and bandwidth of 200MB/s and node
1 has
access latency of 10ns and bandwidth of only 100MB/s. The memory cache
level 1
on both nodes is 10KB, cache line is 8B long with write-back policy and
direct
associativity (whatever that means).
For better future extensibility I'd express these as separate elements,
rather
than attributes to <cell/> element. For instance like this:
<domain>
<vcpu>2</vcpu>
<cpu>
<numa>
<cell id='0' cpus='0,1' memory='1'
unit='GiB'>
<latencies>
<latency type='access' value='5'/>
<bandwidth type='access' unit='MiB'
value='200'/>
</latencies>
<caches>
<cache level='1' associativity='direct'
policy='write-back'>
<size unit='KiB' value='10'/>
<line unit='B' value='8'/>
</cache>
</caches>
</cell>
<cell id='1' memory='1' unit='GiB'
initiator='0'>
<latencies>
<latency type='access' value='10'/>
<bandwidth type='access' unit='MiB'
value='100'/>
</latencies>
<caches>
<cache level='1' associativity='direct'
policy='write-back'>
<size unit='KiB' value='10'/>
<line unit='B' value='8'/>
</cache>
</caches>
</cell>
</numa>
</cpu>
</domain>
Thing is, the @hierarchy argument accepts: memory (referring to whole
memory),
or first-level|second-level|third-level (referring to side caches for each
domain). I haven't figured out yet, how to express the levels in XML yet.
The @data-type argument accepts access|read|write (this is expressed by
@type
attribute to <latency/> and <bandwidth/> elements). Latency and
bandwidth can
be combined with each type: access-latency, read-latency, write-latency,
access-bandwidth, read-bandwidth, write-bandwidth. And these 6 can then be
combined with aforementioned @hierarchy, producing 24 combinations (if I
read
qemu cmd line specs correctly [2]).
What are your thoughts?
Michal
1: https://bugzilla.redhat.com/show_bug.cgi?id=1786303
2:
https://git.qemu.org/?p=qemu.git;a=blob;f=qemu-options.hx;h=d4b73ef60c1d4...