Hi

I would like to ask for comments and propose a possible new libvirt feature.

 

Problem statement:

At present when you boot a vm via libvirt it is possible  to pin vm vCPUs to host CPUs to improve performance in the  guest under certain conditions.

If hostdev interrupts are not pined to a specific core/cpuset suboptimal processing of irqs may occur reducing the performance of both guest and host.

I would like to propose extending libvirt to automatically pin interrupts for hostdev devices if they are present in the guest.

 

By affinitizing interrupts, cache line sharing between the specified interrupt and guest can be achieved. 

If CPU affinity for the guest, as set by the cpuset parameter,

is intelligently chosen to place the guest on the same numa node as the hostdev, cross socket traffic can be mitigated.

As a result, latency which would be introduce if the interrupt processing was scheduled to a non-local numa node CPU can be reduced via interrupt pinning.

 

Proposed change:

·         util/virpci and util/virhostdev will be extended to retrieve IRQ and msi_interupt information from sysfs.

·         util/virinterupt  will be created

·         util/virinterupt  will implement managing interrupt affinity via /proc/irq/x/smp_affinity

·         qemuProcessInitCpuAffinity will be extended to conditionally affinitize hostdev interrupts to vm vCpus when hostdevs are present in the vm definition.

Alternative implementation:

 

In addition to the above changes the hostdev element could be extended to include a cpuset attribute:

·         if the cpuset is auto: the interrupts would be pinned to the same cpuset as the vms vCPU

·         if a cpuset is specified: the interrupts would be affinitized as per the set cpuset

·         if the cpuset is native: the interrupts will not be pinned and the current behaviour will not be changed.

·         If the cpuset attribute is not present either the behaviour of auto or native could be used as a default.

o   Using auto as the default would allow transparent use of the feature.

o   Using native as the default would allow no changes to any existing deployments unless the feature is requested.

Any feedback is welcomed.

Regards

Sean.