Re: [libvirt] [PATCH 1/4] libxl: implement NUMA capabilities reporting

[Moving the conversation on @xen-devel and adding Jan, as that seems more appropriate] [Jan, this came up as I'm implementing some NUMA bits in libvirt but, as you see, the core of Jim's question is purely about Xen] On lun, 2013-07-01 at 16:47 -0600, Jim Fehlig wrote: Which is 8960, if divided by 1024, so at least it's consistent. However... Mmm... Interesting question. I really never paid attention to this... Jan (or anyone else), is that something known and/or expected? I went checking this down in Xen, and here's what I found. total_memory is: info.total_pages/((1 << 20) / vinfo->pagesize) where 'info' is what libxl_get_physinfo() provides. On its turn, libxl_get_physinfo() is xc_physinfo(), which is XEN_SYSCTL_physinfo, which uses total_pages, which is assigned the number of pages, down in __start_xen(), as it results from parsing the E820 map (looking for RAM blocks). OTOH, memsize comes from libxl_get_numainfo(), which is xc_numainfo(), which is XEN_SYSCTL_numainfo, which puts in memsize what node_spanned_pages(<node_id>) says. That seems to come, on a NUMA box, from the parsing of SRAT, and on a non-NUMA box, from just (start_pfn-end_pfn) (in pages, of course). Anyway, on my NUMA box, I see something similar to what Jim sees on a non-NUMA one: # xl info -n ... total_memory : 12285 ... numa_info : node: memsize memfree distances 0: 6144 23 10,20 1: 6720 104 20,10 Where 6144+6720=12864 > 12285 Looking at what Xen says during boot, I see this (the [*], [+], [=] and [|] are mine): (XEN) Xen-e820 RAM map: (XEN) 0000000000000000 - 0000000000096000 (usable) (XEN) 00000000000f0000 - 0000000000100000 (reserved) [*] (XEN) 0000000000100000 - 00000000dbdf9c00 (usable) (XEN) 00000000dbdf9c00 - 00000000dbe4bc00 (ACPI NVS) [+] (XEN) 00000000dbe4bc00 - 00000000dbe4dc00 (ACPI data) [=] (XEN) 00000000dbe4dc00 - 00000000dc000000 (reserved) [|] (XEN) 00000000f8000000 - 00000000fd000000 (reserved) (XEN) 00000000fe000000 - 00000000fed00400 (reserved) (XEN) 00000000fee00000 - 00000000fef00000 (reserved) (XEN) 00000000ffb00000 - 0000000100000000 (reserved) (XEN) 0000000100000000 - 0000000324000000 (usable) ... (XEN) System RAM: 12285MB (12580412kB) And my math says that 12285MB is the sum of the areas marked as (usable), i.e., I guess, what during parsing is recognised as E820_RAM... which makes total sense. A bit below that I have this: (XEN) SRAT: Node 1 PXM 1 0-dc000000 (XEN) SRAT: Node 1 PXM 1 100000000-1a4000000 (XEN) SRAT: Node 0 PXM 0 1a4000000-324000000 Which, after the needed calculations, gives exactly the same results than memsize-s in `xl info -n'. Now, if I add up the [*], [+], [=] and [|] regions above, and then subtract that from node 1's PXMs, I see that node 1 has only ~6141MB of usable RAM, instead of 6720MB. And in fact, 6720-6141=579, just as much as 12864-12285=579. So, if I haven't messed up with the calculations, it looks like that Xen, when reporting to the upper layers the amount of memory it has available, does filter out the non-RAM regions, if this happens via XEN_SYSCTL_physinfo (i.e., by parsing E820), while it does not do that, if this happens via XEN_SYSCTL_numainfo (i.e., by parsing ACPI SRAT). What I'm not sure about is whether or not that was something known/intended and whether or not it is something we should be concerned about. Thanks and Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)