Re: [libvirt] [PATCH v4 2/5] numa: describe siblings distances within cells

On Fri, 6 Oct 2017 08:49:46 -0600 Jim Fehlig <jfehlig(a)suse.com&gt; wrote: > On 09/08/2017 08:47 AM, Wim Ten Have wrote: >> From: Wim ten Have <wim.ten.have(a)oracle.com&gt; >> >> Add libvirtd NUMA cell domain administration functionality to >> describe underlying cell id sibling distances in full fashion >> when configuring HVM guests. > > May I suggest wording this paragraph as: > > Add support for describing sibling vCPU distances within a domain's vNUMA cell > configuration. See below (v5 comment). >> Schema updates are made to docs/schemas/cputypes.rng enforcing domain >> administration to follow the syntax below the numa cell id and >> docs/schemas/basictypes.rng to add "numaDistanceValue". > > I'm not sure this paragraph is needed in the commit message. > >> A minimum value of 10 representing the LOCAL_DISTANCE as 0-9 are >> reserved values and can not be used as System Locality Distance Information. >> A value of 20 represents the default setting of REMOTE_DISTANCE >> where a maximum value of 255 represents UNREACHABLE. >> >> Effectively any cell sibling can be assigned a distance value where >> practically 'LOCAL_DISTANCE <= value <= UNREACHABLE'. >> >> [below is an example of a 4 node setup] >> >> <cpu> >> <numa> >> <cell id='0' cpus='0' memory='2097152' unit='KiB'> >> <distances> >> <sibling id='0' value='10'/> >> <sibling id='1' value='21'/> >> <sibling id='2' value='31'/> >> <sibling id='3' value='41'/> >> </distances> >> </cell> >> <cell id='1' cpus='1' memory='2097152' unit='KiB'> >> <distances> >> <sibling id='0' value='21'/> >> <sibling id='1' value='10'/> >> <sibling id='2' value='31'/> >> <sibling id='3' value='41'/> >> </distances> >> </cell> >> <cell id='2' cpus='2' memory='2097152' unit='KiB'> >> <distances> >> <sibling id='0' value='31'/> >> <sibling id='1' value='21'/> >> <sibling id='2' value='10'/> >> <sibling id='3' value='21'/> >> </distances> >> <cell id='3' cpus='3' memory='2097152' unit='KiB'> >> <distances> >> <sibling id='0' value='41'/> >> <sibling id='1' value='31'/> >> <sibling id='2' value='21'/> >> <sibling id='3' value='10'/> >> </distances> >> </cell> >> </numa> >> </cpu> > > How would this look when having more than one cpu in a cell? I suppose something > like > > <cpu> > <numa> > <cell id='0' cpus='0-3' memory='2097152' unit='KiB'> > <distances> > <sibling id='0' value='10'/> > <sibling id='1' value='10'/> > <sibling id='2' value='10'/> > <sibling id='3' value='10'/> > <sibling id='4' value='21'/> > <sibling id='5' value='21'/> > <sibling id='6' value='21'/> > <sibling id='7' value='21'/> > </distances> > </cell> > <cell id='1' cpus='4-7' memory='2097152' unit='KiB'> > <distances> > <sibling id='0' value='21'/> > <sibling id='1' value='21'/> > <sibling id='2' value='21'/> > <sibling id='3' value='21'/> > <sibling id='4' value='10'/> > <sibling id='5' value='10'/> > <sibling id='6' value='10'/> > <sibling id='7' value='10'/> > </distances> > </cell> > </numa> > </cpu> Nope. That machine seems to make a 2 node vNUMA setup. Where; * NUMA node(0) defined by <cell id='0'> holds 4 (cores) cpus '0-3' with 2GByte of dedicated memory. * NUMA node(1) defined by <cell id='1'> holds 4 (cores) cpus '4-7' with 2GByte of dedicated memory.

<cpu> <numa> <cell id='0' cpus='0-3' memory='2097152' unit='KiB'> <distances> <sibling id='0' value='10'/> <sibling id='1' value='21'/> </distances> </cell> <cell id='1' cpus='4-7' memory='2097152' unit='KiB'> <distances> <sibling id='0' value='21'/> <sibling id='1' value='10'/> </distances> </cell> </numa> </cpu>

Specific configuration would typically report below when examined from within the guest domain; (despite ignorance in this example that it _DOES_ concern a single socket 8 cpu machine). [root@f25 ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 8 Socket(s): 1 *> NUMA node(s): 2 Vendor ID: AuthenticAMD CPU family: 21 Model: 2 Model name: AMD FX-8320E Eight-Core Processor Stepping: 0 CPU MHz: 3210.862 BogoMIPS: 6421.83 Virtualization: AMD-V Hypervisor vendor: Xen Virtualization type: full L1d cache: 16K L1i cache: 64K L2 cache: 2048K L3 cache: 8192K *> NUMA node0 CPU(s): 0-3 *> NUMA node1 CPU(s): 4-7 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c hypervisor lahf_lm svm cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs xop lwp fma4 tbm vmmcall bmi1 arat npt lbrv nrip_save tsc_scale vmcb_clean decodeassists pausefilter [root@f25 ~]# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 node 0 size: 1990 MB node 0 free: 1786 MB node 1 cpus: 4 5 6 7 node 1 size: 1950 MB node 1 free: 1820 MB node distances: node 0 1 0: 10 21 1: 21 10

> In the V3 thread you mentioned "And to reduce even more we could also > remove LOCAL_DISTANCES as they make a constant factor where; (cell_id == > sibling_id)". In the above example cell_id 1 == sibling_id 1, but it is not > LOCAL_DISTANCE. > >> Whenever a sibling id the cell LOCAL_DISTANCE does apply and for any >> sibling id not being covered a default of REMOTE_DISTANCE is used >> for internal computations. > > I'm having a hard time understanding this sentence... Me.2 > I didn't look closely at the patch since I'd like to understand how multi-cpu > cells are handled before doing so. Let me prepare v5. I found a silly error in code being fixed and given above commented confusion like to take a better approach under the commit messages and witin the cover letter.