2:49 a.m.
On Sun, 30 Jun 2024, Tejun Heo wrote:
Hello,
On Sat, Jun 29, 2024 at 08:15:56PM +0200, Mikulas Patocka wrote:
> With 6.5, we get 3600MiB/s; with 6.6 we get 1400MiB/s.
>
> The reason is that virt-manager by default sets up a topology where we
> have 16 sockets, 1 core per socket, 1 thread per core. And that workqueue
> patch avoids moving work items across sockets, so it processes all
> encryption work only on one virtual CPU.
>
> The performance degradation may be fixed with "echo 'system'
> >/sys/module/workqueue/parameters/default_affinity_scope" - but it is
> regression anyway, as many users don't know about this option.
>
> How should we fix it? There are several options:
> 1. revert back to 'numa' affinity
> 2. revert to 'numa' affinity only if we are in a virtual machine
> 3. hack dm-crypt to set the 'numa' affinity for the affected workqueues
> 4. any other solution?
Do you happen to know why libvirt is doing that? There are many other
implications to configuring the system that way and I don't think we want to
design kernel behaviors to suit topology information fed to VMs which can be
arbitrary.
Thanks.
I don't know why. I added users(a)lists.libvirt.org to the CC.
How should libvirt properly advertise "we have 16 threads that are
dynamically scheduled by the host kernel, so the latencies between them
are changing and unpredictable"?
Mikulas
8:48 p.m.
New subject: dm-crypt performance regression due to workqueue changes
On 6/30/24 20:49, Mikulas Patocka wrote:
On Sun, 30 Jun 2024, Tejun Heo wrote:
> Hello,
>
> On Sat, Jun 29, 2024 at 08:15:56PM +0200, Mikulas Patocka wrote:
>
>> With 6.5, we get 3600MiB/s; with 6.6 we get 1400MiB/s.
>>
>> The reason is that virt-manager by default sets up a topology where we
>> have 16 sockets, 1 core per socket, 1 thread per core. And that workqueue
>> patch avoids moving work items across sockets, so it processes all
>> encryption work only on one virtual CPU.
>>
>> The performance degradation may be fixed with "echo 'system'
>>> /sys/module/workqueue/parameters/default_affinity_scope" - but it is
>> regression anyway, as many users don't know about this option.
>>
>> How should we fix it? There are several options:
>> 1. revert back to 'numa' affinity
>> 2. revert to 'numa' affinity only if we are in a virtual machine
>> 3. hack dm-crypt to set the 'numa' affinity for the affected workqueues
>> 4. any other solution?
>
> Do you happen to know why libvirt is doing that? There are many other
> implications to configuring the system that way and I don't think we want to
> design kernel behaviors to suit topology information fed to VMs which can be
> arbitrary.
Firstly, libvirt's not doing anything. It very specifically avoids doing
policy decisions. If something configures vCPUs so that they are in
separate sockets, then we should look at that something. Alternatively,
if "default" configuration does not work for your workflow well,
document recommended configuration.
>
> Thanks.
I don't know why. I added users(a)lists.libvirt.org to the CC.
How should libvirt properly advertise "we have 16 threads that are
dynamically scheduled by the host kernel, so the latencies between them
are changing and unpredictable"?
Libvirt advertises topology of physical CPUs (pCPUS) in so called
capabilities XML (virsh capabilities). For example:
https://libvirt.org/formatcaps.html#examples
(not to be mixed with domain capabilities!)
From there you can see what pCPUs are in the same socket. And regarding
latency - unless you're doing real time, latency is unpredictable even
within single socket, isn't it.
Michal
9:25 p.m.
New subject: dm-crypt performance regression due to workqueue changes
On Mon, Jul 01, 2024 at 02:48:07PM +0200, Michal Prívozník wrote:
On 6/30/24 20:49, Mikulas Patocka wrote:
>
>
> On Sun, 30 Jun 2024, Tejun Heo wrote:
>
>> Hello,
>>
>> On Sat, Jun 29, 2024 at 08:15:56PM +0200, Mikulas Patocka wrote:
>>
>>> With 6.5, we get 3600MiB/s; with 6.6 we get 1400MiB/s.
>>>
>>> The reason is that virt-manager by default sets up a topology where we
>>> have 16 sockets, 1 core per socket, 1 thread per core. And that workqueue
>>> patch avoids moving work items across sockets, so it processes all
>>> encryption work only on one virtual CPU.
>>>
>>> The performance degradation may be fixed with "echo 'system'
>>>> /sys/module/workqueue/parameters/default_affinity_scope" - but it
is
>>> regression anyway, as many users don't know about this option.
>>>
>>> How should we fix it? There are several options:
>>> 1. revert back to 'numa' affinity
>>> 2. revert to 'numa' affinity only if we are in a virtual machine
>>> 3. hack dm-crypt to set the 'numa' affinity for the affected
workqueues
>>> 4. any other solution?
>>
>> Do you happen to know why libvirt is doing that? There are many other
>> implications to configuring the system that way and I don't think we want
to
>> design kernel behaviors to suit topology information fed to VMs which can be
>> arbitrary.
Firstly, libvirt's not doing anything. It very specifically avoids doing
policy decisions. If something configures vCPUs so that they are in
separate sockets, then we should look at that something. Alternatively,
if "default" configuration does not work for your workflow well,
document recommended configuration.
Actually in this particular case, it is strictly speaking libvirt.
If the guest XML config does not mention any <topology> info, then
libvirt explicitly tells QEMU to set sockets=N,cores=1,threads=1.
That matches QEMU's own historical built-in default topology.
None the less, my advice for mgmt applications using libvirt would
likely be to explicitly request sockets=1,cores=N,threads=1. This
is because it gives slightly better compatibility with unpleasant
software that applies licensing / subscription rules that penalize
use of many sockets, while being happy with any number of cores.
Either way though, the topology is a lie when the guest CPUs
are not pinned to host CPUs, so making performance decisions based
on this is unlikely to yield the desired results. Historically the
cores vs sockets distinction hasn't seemed to make much difference
to guest OS performance, as the OS' haven't made significant
decisions on this axis. Exposing threads != 1 though has always been
a big no though, unless strictly pinning 1:1 guest:host CPUs, as that
has had notable impacts on scheduling decisions.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:40 p.m.
New subject: dm-crypt performance regression due to workqueue changes
On Mon, 1 Jul 2024, Daniel P. Berrangé wrote:
On Mon, Jul 01, 2024 at 02:48:07PM +0200, Michal Prívozník wrote:
> On 6/30/24 20:49, Mikulas Patocka wrote:
> >
> >
> > On Sun, 30 Jun 2024, Tejun Heo wrote:
> >>
> >> Do you happen to know why libvirt is doing that? There are many other
> >> implications to configuring the system that way and I don't think we
want to
> >> design kernel behaviors to suit topology information fed to VMs which can
be
> >> arbitrary.
>
> Firstly, libvirt's not doing anything. It very specifically avoids doing
> policy decisions. If something configures vCPUs so that they are in
> separate sockets, then we should look at that something. Alternatively,
> if "default" configuration does not work for your workflow well,
> document recommended configuration.
Actually in this particular case, it is strictly speaking libvirt.
If the guest XML config does not mention any <topology> info, then
libvirt explicitly tells QEMU to set sockets=N,cores=1,threads=1.
That matches QEMU's own historical built-in default topology.
None the less, my advice for mgmt applications using libvirt would
likely be to explicitly request sockets=1,cores=N,threads=1. This
is because it gives slightly better compatibility with unpleasant
software that applies licensing / subscription rules that penalize
use of many sockets, while being happy with any number of cores.
Either way though, the topology is a lie when the guest CPUs
are not pinned to host CPUs, so making performance decisions based
on this is unlikely to yield the desired results. Historically the
cores vs sockets distinction hasn't seemed to make much difference
to guest OS performance, as the OS' haven't made significant
decisions on this axis. Exposing threads != 1 though has always been
a big no though, unless strictly pinning 1:1 guest:host CPUs, as that
has had notable impacts on scheduling decisions.
With regards,
Daniel
I think there should be some way how to tell the guest kernel "the vCPUs
are free-floating, the topology is a lie", so that it can stop making
incorrect decisions based on the fake topology.
Mikulas
9:08 p.m.
New subject: dm-crypt performance regression due to workqueue changes
On Sun, Jun 30, 2024 at 08:49:48PM +0200, Mikulas Patocka wrote:
On Sun, 30 Jun 2024, Tejun Heo wrote:
> Hello,
>
> On Sat, Jun 29, 2024 at 08:15:56PM +0200, Mikulas Patocka wrote:
>
> > With 6.5, we get 3600MiB/s; with 6.6 we get 1400MiB/s.
> >
> > The reason is that virt-manager by default sets up a topology where we
> > have 16 sockets, 1 core per socket, 1 thread per core. And that workqueue
> > patch avoids moving work items across sockets, so it processes all
> > encryption work only on one virtual CPU.
> > The performance degradation may be fixed with "echo 'system'
> > >/sys/module/workqueue/parameters/default_affinity_scope" - but it is
> > regression anyway, as many users don't know about this option.
> >
> > How should we fix it? There are several options:
> > 1. revert back to 'numa' affinity
> > 2. revert to 'numa' affinity only if we are in a virtual machine
> > 3. hack dm-crypt to set the 'numa' affinity for the affected
workqueues
> > 4. any other solution?
>
> Do you happen to know why libvirt is doing that? There are many other
> implications to configuring the system that way and I don't think we want to
> design kernel behaviors to suit topology information fed to VMs which can be
> arbitrary.
>
> Thanks.
I don't know why. I added users(a)lists.libvirt.org to the CC.
How should libvirt properly advertise "we have 16 threads that are
dynamically scheduled by the host kernel, so the latencies between them
are changing and unpredictable"?
NB, libvirt is just control plane, the actual virtual hardware exposed
is implemented across QEMU and the KVM kernel mod. Guest CPU topology
and/or NUMA cost information is the responsibility of QEMU.
When QEMU's virtual CPUs are floating freely across host CPUs there's
no perfect answer. The host admin needs to make a tradeoff in their
configuration
They can optimize for density, by allowing guest CPUs to float freely
and allow CPU overcommit against host CPUs, and the guest CPU topology
is essentially a lie.
They can optimize for predictable performance, by strictly pinning
guest CPUs 1:1 to host CPUs, and minimize CPU overcommit, and have
the guest CPU topology 1:1 match the host CPU topology.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:42 p.m.
New subject: dm-crypt performance regression due to workqueue changes
On Mon, 1 Jul 2024, Daniel P. Berrangé wrote:
On Sun, Jun 30, 2024 at 08:49:48PM +0200, Mikulas Patocka wrote:
>
>
> On Sun, 30 Jun 2024, Tejun Heo wrote:
>
> > Do you happen to know why libvirt is doing that? There are many other
> > implications to configuring the system that way and I don't think we want
to
> > design kernel behaviors to suit topology information fed to VMs which can be
> > arbitrary.
> >
> > Thanks.
>
> I don't know why. I added users(a)lists.libvirt.org to the CC.
>
> How should libvirt properly advertise "we have 16 threads that are
> dynamically scheduled by the host kernel, so the latencies between them
> are changing and unpredictable"?
NB, libvirt is just control plane, the actual virtual hardware exposed
is implemented across QEMU and the KVM kernel mod. Guest CPU topology
and/or NUMA cost information is the responsibility of QEMU.
When QEMU's virtual CPUs are floating freely across host CPUs there's
no perfect answer. The host admin needs to make a tradeoff in their
configuration
They can optimize for density, by allowing guest CPUs to float freely
and allow CPU overcommit against host CPUs, and the guest CPU topology
is essentially a lie.
They can optimize for predictable performance, by strictly pinning
guest CPUs 1:1 to host CPUs, and minimize CPU overcommit, and have
the guest CPU topology 1:1 match the host CPU topology.
With regards,
Daniel
The problem that we have here is that the commit
63c5484e74952f60f5810256bd69814d167b8d22 ("workqueue: Add multiple
affinity scopes and interface to select them") changes the behavior of
unbound workqueues, so that work items are only executed on CPUs that
share last level cache with the task that submitted them.
If there are 16 virtual CPUs that are freely floating across physical
CPUs, virt-manager by default selects a topology where it advertises 16
sockets, 1 CPU per socket, 1 thread per CPU. The result is that the
unbound workqueues are no longer unbound, they can't move work across
sockets and they are bound to just one virtual CPU, causing dm-crypt
performance degradation. (the crypto operations are no longer
parallelized).
Whose bug is this? Is it a bug in virt-manager because it advertises
invalid topology? Is this a bug in that patch 63c5484e7495 because it
avoids moving work items across sockets?
Mikulas
9:52 p.m.
New subject: dm-crypt performance regression due to workqueue changes
On Mon, Jul 01, 2024 at 03:42:29PM +0200, Mikulas Patocka wrote:
On Mon, 1 Jul 2024, Daniel P. Berrangé wrote:
> On Sun, Jun 30, 2024 at 08:49:48PM +0200, Mikulas Patocka wrote:
> >
> >
> > On Sun, 30 Jun 2024, Tejun Heo wrote:
> >
> > > Do you happen to know why libvirt is doing that? There are many other
> > > implications to configuring the system that way and I don't think we
want to
> > > design kernel behaviors to suit topology information fed to VMs which can
be
> > > arbitrary.
> > >
> > > Thanks.
> >
> > I don't know why. I added users(a)lists.libvirt.org to the CC.
> >
> > How should libvirt properly advertise "we have 16 threads that are
> > dynamically scheduled by the host kernel, so the latencies between them
> > are changing and unpredictable"?
>
> NB, libvirt is just control plane, the actual virtual hardware exposed
> is implemented across QEMU and the KVM kernel mod. Guest CPU topology
> and/or NUMA cost information is the responsibility of QEMU.
>
> When QEMU's virtual CPUs are floating freely across host CPUs there's
> no perfect answer. The host admin needs to make a tradeoff in their
> configuration
>
> They can optimize for density, by allowing guest CPUs to float freely
> and allow CPU overcommit against host CPUs, and the guest CPU topology
> is essentially a lie.
>
> They can optimize for predictable performance, by strictly pinning
> guest CPUs 1:1 to host CPUs, and minimize CPU overcommit, and have
> the guest CPU topology 1:1 match the host CPU topology.
The problem that we have here is that the commit
63c5484e74952f60f5810256bd69814d167b8d22 ("workqueue: Add multiple
affinity scopes and interface to select them") changes the behavior of
unbound workqueues, so that work items are only executed on CPUs that
share last level cache with the task that submitted them.
If there are 16 virtual CPUs that are freely floating across physical
CPUs, virt-manager by default selects a topology where it advertises 16
sockets, 1 CPU per socket, 1 thread per CPU. The result is that the
unbound workqueues are no longer unbound, they can't move work across
sockets and they are bound to just one virtual CPU, causing dm-crypt
performance degradation. (the crypto operations are no longer
parallelized).
Whose bug is this? Is it a bug in virt-manager because it advertises
invalid topology? Is this a bug in that patch 63c5484e7495 because it
avoids moving work items across sockets?
It is hard to call it is a bug in anything. The Linux patch is reasonable
in honouring the CPU topology. The hypervisor is reasonable to exposing
sockets=N,cores=1 as there's no right answer for the topology choice, and
it has no idea how it may or may not impact guest OS behaviour or perf.
Letting CPUs float freely is sensible default behaviour too. All of them
conspire to have a perf impact here, but the deployment is not seeking
to maximise performance, rather to maximise flexibility & density.
None the less, I'd suggest that virt-manager should be a explicitly
asking for sockets=1,cores=N, as that has broader guest OS compatibility.
By chance that would also help this scenario, but that woudn't be a
driving factor as we can't pick defaults based on the needs of particular
versions of a particular guest kernel.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1:25 a.m.
New subject: dm-crypt performance regression due to workqueue changes
Hello,
On Mon, Jul 01, 2024 at 02:52:19PM +0100, Daniel P. Berrangé wrote:
...
None the less, I'd suggest that virt-manager should be a
explicitly
asking for sockets=1,cores=N, as that has broader guest OS compatibility.
+1. Multiple sockets is pretty uncommon and often comes with significant
performance implications - e.g. if VM is also splitting memory into N nodes,
that can lead to significant higher overhead during reclaim due to node
imbalances and premature OOMs. If the reported topology is not real, it
makes a lot more sense to keep it basic.
Thanks.
--
tejun
125
days inactive
126
days old
7 comments
4 participants
participants (4)
-
Daniel P. Berrangé -
Michal Prívozník -
Mikulas Patocka -
Tejun Heo