9:11 a.m.
On Mon, Dec 05, 2016 at 04:12:22PM +0000, Feng, Shaohe wrote:
Hi all:
As we are know Intel® Xeon phi targets high-performance computing and other
parallel workloads.
Now qemu has supported phi virtualization，it is time for libvirt to support
phi.
Can you provide pointer to the relevant QEMU changes.
Different from the traditional X86 server, There is a special numa
node with
Multi-Channel DRAM (MCDRAM) on Phi, but without any CPU .
Now libvirt requires nonempty cpus argument for NUMA node, such as.
<numa>
<cell id='0' cpus='0-239' memory='80'
unit='GiB'/>
<cell id='1' cpus='240-243' memory='16'
unit='GiB'/>
</numa>
In order to support phi virtualization, libvirt needs to allow a numa cell
definition without 'cpu' attribution.
Such as:
<numa>
<cell id='0' cpus='0-239' memory='80'
unit='GiB'/>
<cell id='1' memory='16' unit='GiB'/>
</numa>
When a cell without 'cpu', qemu will allocate memory by default MCDRAM instead of
DDR.
There's separate concepts at play which your description here is
mixing up.
First is the question of whether the guest NUMA node can be created
with only RAM or CPUs, or a mix of both.
Second is the question of what kind of host RAM (MCDRAM vs DDR) is
used as the backing store for the guest
These are separate configuration items which don't need to be
conflated in libvirt. ie we should be able to create a guest
with a node containing only memory, and back that by DDR on
the host. Conversely we should be able to create a guest
with a node containing memory + cpus and back that by MCDRAM
on the host (even if that means the vCPUs will end up on a
different host node from its RAM)
On the first point, there still appears to be some brokness
in either QEMU or Linux wrt configuration of virtual NUMA
where either cpus or memory are absent from nodes.
eg if I launch QEMU with
-numa node,nodeid=0,cpus=0-3,mem=512
-numa node,nodeid=1,mem=512
-numa node,nodeid=2,cpus=4-7
-numa node,nodeid=3,mem=512
-numa node,nodeid=4,mem=512
-numa node,nodeid=5,cpus=8-11
-numa node,nodeid=6,mem=1024
-numa node,nodeid=7,cpus=12-15,mem=1024
then the guest reports
# numactl --hardware
available: 6 nodes (0,3-7)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
node 0 size: 487 MB
node 0 free: 230 MB
node 3 cpus: 12 13 14 15
node 3 size: 1006 MB
node 3 free: 764 MB
node 4 cpus:
node 4 size: 503 MB
node 4 free: 498 MB
node 5 cpus:
node 5 size: 503 MB
node 5 free: 499 MB
node 6 cpus:
node 6 size: 503 MB
node 6 free: 498 MB
node 7 cpus:
node 7 size: 943 MB
node 7 free: 939 MB
so its pushed all the CPUs from nodes without RAM into the
first node, and moved CPUs from the 7th node into the 3rd
node.
So before considering MCDRAM / Phi, we need to fix this more
basic NUMA topology setup.
Now here I'd like to discuss these questions:
1. This feature is only for Phi at present, but we
will check Phi platform for CPU-less NUMA node.
The NUMA node without CPU indicates MCDRAM node.
We should not assume such semantics - it is a concept
that is specific to particular Intel x86_64 CPUs. We
need to consider that other architectures may have
nodes without CPUs that are backed by normal DDR.
IOW, we shoud be explicit about presence of MCDRAM
in the host.
And for now MCDRAM is available only for PHI.
However, there is no reason that any other platform
couldn’t define CPU-less NUMA node using libvirt, so
there is no reason to check if PHI is used or not.
2. Type of memory of CPU-less NUMA node will not be
checked during machine creation/configuration step.
There is no reliable way to distinguish if the node
is MCDRAM or regular DDR. This step is not concerned
with type of the memory, only with NUMA assignment.
If we can't distinguish MCDRAM from DDR that's a problem
for apps, given your next point about MCDRAM not supporting
over commit.
3. Unlike traditional memory assign to a VM, MCDRAM do not
support over commit
If the memory of a virtual NUMA node is going to be
explicitly bound to physical NUMA node then it shouldn’t
exceed the size of its corresponding NUMA node, doesn’t
matter if it is MCDRAM or DDR.
It is valid to bind guests to NUMA nodes and still have
memory over commit, so we do need to know if a host node
is using MCDRAM or DDR, so apps can determine whether
that node supports over commit or not.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|