How to find the actually schedulable CPUs in a Linux + cgroups setup?
- First Post
- Replies
- Stats
-
Go to
- ----- 2025 -----
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2008 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2007 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2006 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2005 -----
- December
Hi,
Right now, looking at the QEMU driver, libvirt supports listing all of
the present CPUs on the hypervisor via /sys/devices/system/cpu/present.
For systems where cgroups with exclusive CPU bits are configured,
however, there might be CPUs that are not availabe to libvirt. This
leads to situations where a domain cannot be started because of an
invalid cpuset attribute (using a CPU with an exclusive bit set).
My initial idea was to refactor functions like virNodeGetCPUMap() to
take into account cgroup limits. danpb mentioned in #virt (thanks
danpb!) that libvirtd might run in a cgroup that is not the one in which
VMs will run (like on systemd hosts where libvirtd will run under
/sys/fs/cgroup/system.slice/, and VMs under /sys/cgroup/machine.slice/),
so just reading hardcoded cgroup files is not a good idea.
Is it possible for libvirt to figure out what the actually schedulable
CPUs are? Ideally users can then query this information via `virsh
nodeinfo` or `virsh capabilities`.
---
Regards,
Biser
Attachments:
- attachment.html (text/html — 1.1 KB)
Monday, 14 April
Mon, 14 Apr
3:46 p.m.
New subject: How to find the actually schedulable CPUs in a Linux + cgroups
setup?
On Fri, Apr 04, 2025 at 04:32:42PM +0300, Biser Milanov via Devel wrote:
Hi,
Right now, looking at the QEMU driver, libvirt supports listing all of
the present CPUs on the hypervisor via /sys/devices/system/cpu/present.
For systems where cgroups with exclusive CPU bits are configured,
however, there might be CPUs that are not availabe to libvirt. This
leads to situations where a domain cannot be started because of an
invalid cpuset attribute (using a CPU with an exclusive bit set).
My initial idea was to refactor functions like virNodeGetCPUMap() to
take into account cgroup limits. danpb mentioned in #virt (thanks
danpb!) that libvirtd might run in a cgroup that is not the one in which
VMs will run (like on systemd hosts where libvirtd will run under
/sys/fs/cgroup/system.slice/, and VMs under /sys/cgroup/machine.slice/),
so just reading hardcoded cgroup files is not a good idea.
Yes, there are several reasons why libvirt is trying to schedule the VM
on all CPUs and rather provides ways to limit that selection.
Having said that, we also already subtract isolated CPUs from the full
list (if there is no pinning set for the VM) since commit da95bcb6b2d9
https://gitlab.com/libvirt/libvirt/-/commit/da95bcb6b2d9b04958e0f26032028...
However it is a bit trickier with exclusive CPUs. AFAIU isolated CPUs
are static between reboots so it is easier to know what to subtract.
So I would rather vouch for letting the users decide based on what's
the current host setup, which they should, ideally, know beforehand.
Is it possible for libvirt to figure out what the actually
schedulable
CPUs are? Ideally users can then query this information via `virsh
nodeinfo` or `virsh capabilities`.
If that is not known for some reason, then we could provide that
information, but is there a reliable way to get all the non-exclusive
CPUs that could be used for a VM? I'm worried about two things:
1) the information gathering being racy (by the time we check one
cgroup there might be a change in another)
2) there are more than one file per cgroup to check in order to make it
correct and the logic behind it is not too straightforward.
---
Regards,
Biser
4
days inactive
14
days old
1 comments
2 participants
tags (0)
participants (2)
-
Biser Milanov -
Martin Kletzander