How to find the actually schedulable CPUs in a Linux + cgroups setup?
Hi, Right now, looking at the QEMU driver, libvirt supports listing all of the present CPUs on the hypervisor via /sys/devices/system/cpu/present. For systems where cgroups with exclusive CPU bits are configured, however, there might be CPUs that are not availabe to libvirt. This leads to situations where a domain cannot be started because of an invalid cpuset attribute (using a CPU with an exclusive bit set). My initial idea was to refactor functions like virNodeGetCPUMap() to take into account cgroup limits. danpb mentioned in #virt (thanks danpb!) that libvirtd might run in a cgroup that is not the one in which VMs will run (like on systemd hosts where libvirtd will run under /sys/fs/cgroup/system.slice/, and VMs under /sys/cgroup/machine.slice/), so just reading hardcoded cgroup files is not a good idea. Is it possible for libvirt to figure out what the actually schedulable CPUs are? Ideally users can then query this information via `virsh nodeinfo` or `virsh capabilities`. --- Regards, Biser
On Fri, Apr 04, 2025 at 04:32:42PM +0300, Biser Milanov via Devel wrote:
Hi,
Right now, looking at the QEMU driver, libvirt supports listing all of the present CPUs on the hypervisor via /sys/devices/system/cpu/present. For systems where cgroups with exclusive CPU bits are configured, however, there might be CPUs that are not availabe to libvirt. This leads to situations where a domain cannot be started because of an invalid cpuset attribute (using a CPU with an exclusive bit set).
My initial idea was to refactor functions like virNodeGetCPUMap() to take into account cgroup limits. danpb mentioned in #virt (thanks danpb!) that libvirtd might run in a cgroup that is not the one in which VMs will run (like on systemd hosts where libvirtd will run under /sys/fs/cgroup/system.slice/, and VMs under /sys/cgroup/machine.slice/), so just reading hardcoded cgroup files is not a good idea.
Yes, there are several reasons why libvirt is trying to schedule the VM on all CPUs and rather provides ways to limit that selection. Having said that, we also already subtract isolated CPUs from the full list (if there is no pinning set for the VM) since commit da95bcb6b2d9 https://gitlab.com/libvirt/libvirt/-/commit/da95bcb6b2d9b04958e0f2603202801d... However it is a bit trickier with exclusive CPUs. AFAIU isolated CPUs are static between reboots so it is easier to know what to subtract. So I would rather vouch for letting the users decide based on what's the current host setup, which they should, ideally, know beforehand.
Is it possible for libvirt to figure out what the actually schedulable CPUs are? Ideally users can then query this information via `virsh nodeinfo` or `virsh capabilities`.
If that is not known for some reason, then we could provide that information, but is there a reliable way to get all the non-exclusive CPUs that could be used for a VM? I'm worried about two things: 1) the information gathering being racy (by the time we check one cgroup there might be a change in another) 2) there are more than one file per cgroup to check in order to make it correct and the logic behind it is not too straightforward.
--- Regards, Biser
participants (2)
-
Biser Milanov -
Martin Kletzander