On Mon, Aug 18, 2014 at 07:01:40PM +0000, Mooney, Sean K wrote:
Hi
I would like to ask for comments and propose a possible new libvirt feature.
Hi, could you convince your e-mail agent to wrap long lines?
Problem statement:
At present when you boot a vm via libvirt it is possible to pin vm vCPUs to host CPUs to
improve performance in the guest under certain conditions.
If hostdev interrupts are not pined to a specific core/cpuset suboptimal processing of
irqs may occur reducing the performance of both guest and host.
I would like to propose extending libvirt to automatically pin interrupts for hostdev
devices if they are present in the guest.
By affinitizing interrupts, cache line sharing between the specified interrupt and guest
can be achieved.
If CPU affinity for the guest, as set by the cpuset parameter,
is intelligently chosen to place the guest on the same numa node as the hostdev, cross
socket traffic can be mitigated.
As a result, latency which would be introduce if the interrupt processing was scheduled to
a non-local numa node CPU can be reduced via interrupt pinning.
Proposed change:
* util/virpci and util/virhostdev will be extended to retrieve IRQ and
msi_interupt information from sysfs.
* util/virinterupt will be created
* util/virinterupt will implement managing interrupt affinity via
/proc/irq/x/smp_affinity
Nice that this will be abstracted out in a separate file, although if
it's not huge, it can be part of something else.
* qemuProcessInitCpuAffinity will be extended to conditionally
affinitize hostdev interrupts to vm vCpus when hostdevs are present in the vm definition.
So it will only affect hostdevs, OK, that means there should be less
(or no) disadvantages). Although beware that hostdevs can be
plugged/unplugged, number of vCPUs can be changed and, most
importantly, the affinities (either set with sched_set_affinity or
using cgroups) can be changed during the lifetime of the VM, and the
smp_affinity of each IRQ should track such changes. Needless to say
the affinity should be restored after the machine dies/is turned off,
etc., not just on hostdev unplug.
Alternative implementation:
In addition to the above changes the hostdev element could be extended to include a cpuset
attribute:
* if the cpuset is auto: the interrupts would be pinned to the same cpuset as the
vms vCPU
* if a cpuset is specified: the interrupts would be affinitized as per the set
cpuset
* if the cpuset is native: the interrupts will not be pinned and the current
behaviour will not be changed.
* If the cpuset attribute is not present either the behaviour of auto or native
could be used as a default.
o Using auto as the default would allow transparent use of the feature.
I like defaulting to auto also because the transparent use will have
effect only in existing deployments with vCPU pinning.
o Using native as the default would allow no changes to any existing
deployments unless the feature is requested.
Although this version might be preferred by others. This can,
however, be discussed (and changed) during reviews.
Any feedback is welcomed.
Regards
Sean.
--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare
This e-mail and any attachments may contain confidential material for the sole use of the
intended recipient(s). Any review or distribution by others is strictly prohibited. If you
are not the intended recipient, please contact the sender and delete all copies.
Since this mailing-list is public and archived, such disclaimer is
unenforceable. As an upstream project, the development is done in the
open. Such statements as the above one may be the cause of nobody
replying for some time to your e-mail.
The statement itself can be understood that even delivering the mail
through a list is prohibited. Please consider removing such
statements in following e-mails (or use private e-mail address if the
disclaimer is enforced by your employer) as keeping them may result in
the rest of the community not being able to communicate with you.
Martin