Re: cpugroups for hyperv hypervisor
On Wed, Jun 26, 2024 at 09:18:37AM -0500, Praveen K Paladugu wrote:
Hey folks,
My team is working on exposing `cpugroups` to Libvirt while using 'hyperv'
hypervisor with cloud-hypervisor(VMM). cpugroups are relevant in a specific
configuration of hyperv called 'minroot'. In Minroot configuration,
hypervisor artificially restricts Dom0 to run on a subset of cpus (Logical
Processors). The rest of the cpus can be assigned to guests.
cpugroups manage the CPUs assigned to guests and their scheduling
properties. Initially this looks similar to `cpuset` (in cgroups), but the
controls available with cpugroups don't map easily to those in cgroups. For
example:
* "IdleLPs" are the number of Logical Processors in a cpugroup, that should
be reserved to a guest even if they are idle
Are you saying that "IdleLPs" are host CPUs that are reserved for
a guest, but which are NOT currently going to be used for running
any virtual guest CPUs ?
At what point do IdleLPs become used (non-idle) by the guest ?
* "SchedulingPriority", the priority(values between 0..7)
with which to
schedule CPUs in a cpugroup.
We currently have
<vcpusched vcpus='0-4,^3' scheduler='fifo'
priority='1'/>
and 'SchedulePriority' would be conceptually mapping to the 'priority'
value.
It sounds like you're saying that the priority applies to /all/
CPUs in the cpugroup. IOW, if we were to re-use <vcpusched> for
this, we would have to require that the 'vcpus' mask always
covers every CPU in the cpugroup.
It is probably better to just declare a new global element:
<cputune>
<priority>0..7</priority>
</cputune>
since we've got precedent there with global elements for
<shares>, <period>, <quota>, etc setting overall VM policy,
which can option be refined per-vCPU by other elements.
As controls like above don't easily map to anything in cgroups,
using a
driver specific element in Domain xml, to configure cpugroups seems like a
right approach. For example:
I think our general view is that tunable parameters in general are
almost entirely driver specific.
We provide a generic API framework for tunables using the virTypedParameter
arrays. The named tunables listed within the parameter array though, will
generally be different per-driver. Similarly we have the general <cputune>
element, but stuff within that is often per-driver.
If there are some parameters which are common to many drivers that's a
bonus, but I wouldn't say that is a required expectation.
IOW, I don't expect the cloud hypervisor driver to use a custom XML
namespace for this tasks. We should define new XML elements and/or
virTypedParameter constant names as needed, and re-use existing stuff
where sensible.
<ch:cpugroups>
<idle_lps value='4'/>
<scheduling_priority value='6'/>
</ch:cpugroups>
As cpugroups is only relevant while using minroot configuration on hyperv, I
don't see any value in generalizing this setting. So, having some "ch"
driver specific settings seems like a good approach to implement this
feature.
Question1: Do you see any concerns with this approach?
The cpugroup settings can be applied/modified using sysfs interface or using
a cmdline tool on the host. I see Libvirt uses both these mechanisms for
various use cases. But, given a choice, sysfs based interface seems like a
simpler approach to me. With sysfs interface Libvirt does not have to take
install time dependencies on new tools
Question2: Of "sysfs" vs "cmdline tool" which is preferred, given a
choice?
Directly using sysfs is preferrable. It has lower overhead, and we can see
directly what fails allowing clearer error reporting when needed. sysfs is
simple enough that spawning a cmdline tool doesn't reduce our work, and if
anything increases the work.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|