Re: [libvirt PATCH] docs: add kbase entry showing KVM real time guest config

There are many different settings that required to config a KVM guest for real time, low latency workoads. The documentation included here is based on guidance developed & tested by the Red Hat KVM real time team. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- docs/kbase.html.in | 3 + docs/kbase/kvm-realtime.rst | 213 ++++++++++++++++++++++++++++++++++++ 2 files changed, 216 insertions(+) create mode 100644 docs/kbase/kvm-realtime.rst diff --git a/docs/kbase.html.in b/docs/kbase.html.in index c586e0f676..e663ca525f 100644 --- a/docs/kbase.html.in +++ b/docs/kbase.html.in @@ -36,6 +36,9 @@ <dt><a href="kbase/virtiofs.html">Virtio-FS</a></dt> <dd>Share a filesystem between the guest and the host</dd> + + <dt><a href="kbase/kvm-realtime.html">KVM real time</a></dt> + <dd>Run real time workloads in guests on a KVM hypervisor</dd> </dl> </div> diff --git a/docs/kbase/kvm-realtime.rst b/docs/kbase/kvm-realtime.rst new file mode 100644 index 0000000000..ac6102879b --- /dev/null +++ b/docs/kbase/kvm-realtime.rst @@ -0,0 +1,213 @@ +========================== +KVM Real Time Guest Config +========================== + +.. contents:: + +The KVM hypervisor is capable of running real time guest workloads. This page +describes the key pieces of configuration required in the domain XML to achieve +the low latency needs of real time workloads. + +For the most part, configuration of the host OS is out of scope of this +documentation. Refer to the operating system vendor's guidance on configuring +the host OS and hardware for real time. Note in particular that the default +kernel used by most Linux distros is not suitable for low latency real time and +must be replaced by an special kernel build.

+ + +Host partitioning plan +====================== + +Running real time workloads requires carefully partitioning up the host OS +resources, such that the KVM / QEMU processes are strictly separated from any +other workload running on the host, both userspace processes and kernel threads. + +As such, some subset of host CPUs need to be reserved exclusively for running +KVM guests. This requires that the host kernel be booted using the ``isolcpus`` +kernel command line parameter. This parameter removes a set of CPUs from the +schedular, such that that no kernel threads or userspace processes will ever get

+placed on those CPUs automatically. KVM guests are then manually placed onto +these CPUs. + +Deciding which host CPUs to reserve for real time requires understanding of the +guest workload needs and balancing with the host OS needs. The trade off will +also vary based on the physical hardware available. + +For the sake of illustration, this guide will assume a physical machine with two +NUMA nodes, each with 2 sockets and 4 cores, giving a total of 16 CPUs on the

+host. Furthermore, it is assumed that hyperthreading is either not supported or +has been disabled in the BIOS, since it is incompatible with real time. Each +NUMA node is assumed to have 32 GB of RAM, giving 64 GB total for the host. + +It is assumed that 2 CPUs in each NUMA node are reserved for the host OS, with +the remaining 6 CPUs available for KVM real time. With this in mind, the host +kernel should have booted with ``isolcpus=2-7,10,15`` to reserve CPUs.

+ +To maximise efficiency of page table lookups for the guest, the host needs to be +configured with most RAM exposed as huge pages, ideally 1 GB sized. 6 GB of RAM +in each NUMA node will be reserved for general host OS usage as normal sized +pages, leaving 26 GB for KVM usage as huge pages. + +Once huge pages are reserved on the hypothetical machine, the ``virsh +capabilities`` command output is expected to look approximately like: + +:: + + <topology> + <cells num='2'> + <cell id='0'> + <memory unit='KiB'>33554432</memory> + <pages unit='KiB' size='4'>1572864</pages> + <pages unit='KiB' size='2048'>0</pages> + <pages unit='KiB' size='1048576'>26</pages> + <distances> + <sibling id='0' value='10'/> + <sibling id='1' value='21'/> + </distances> + <cpus num='8'> + <cpu id='0' socket_id='0' core_id='0' siblings='0'/> + <cpu id='1' socket_id='0' core_id='1' siblings='1'/> + <cpu id='2' socket_id='0' core_id='2' siblings='2'/> + <cpu id='3' socket_id='0' core_id='3' siblings='3'/> + <cpu id='4' socket_id='1' core_id='0' siblings='4'/> + <cpu id='5' socket_id='1' core_id='1' siblings='5'/> + <cpu id='6' socket_id='1' core_id='2' siblings='6'/> + <cpu id='7' socket_id='1' core_id='3' siblings='7'/> + </cpus> + </cell> + <cell id='1'> + <memory unit='KiB'>33554432</memory> + <pages unit='KiB' size='4'>1572864</pages> + <pages unit='KiB' size='2048'>0</pages> + <pages unit='KiB' size='1048576'>26</pages> + <distances> + <sibling id='0' value='21'/> + <sibling id='1' value='10'/> + </distances> + <cpus num='8'> + <cpu id='8' socket_id='0' core_id='0' siblings='8'/> + <cpu id='9' socket_id='0' core_id='1' siblings='9'/> + <cpu id='10' socket_id='0' core_id='2' siblings='10'/> + <cpu id='11' socket_id='0' core_id='3' siblings='11'/> + <cpu id='12' socket_id='1' core_id='0' siblings='12'/> + <cpu id='13' socket_id='1' core_id='1' siblings='13'/> + <cpu id='14' socket_id='1' core_id='2' siblings='14'/> + <cpu id='15' socket_id='1' core_id='3' siblings='15'/> + </cpus> + </cell> + </cells> + </topology> + +Be aware that CPU ID numbers are not always allocated sequentially as shown +here. It is not unusual to see IDs interleaved between sockets on the two NUMA +nodes, such that ``0-3,8-11`` are be on the first node and ``4-7,12-15`` are on

+the second node. Carefully check the ``virsh capabilities`` output to determine +the CPU ID numbers when configiring both ``isolcpus`` and the guest ``cpuset`` +values. + +Guest configuration +=================== + +What follows is an overview of the key parts of the domain XML that need to be +configured to achieve low latency for real time workflows. The following example +will assume a 4 CPU guest, requiring 16 GB of RAM. It is intended to be placed +on the second host NUMA node. + +CPU configuration +----------------- + +Real time KVM guests intended to run Linux should have a minimum of 2 CPUs. +One vCPU is for running non-real time processes and performing I/O. The other +vCPUs will run real time applications. Some non-Linux OS may not require a +special non-real time CPU to be available, in which case the 2 CPU minimum would +not apply. + +Each guest CPU, even the non-real time one, needs to be pinned to a dedicated +host core that is in the `isolcpus` reserved set. The QEMU emulator threads +also need to be pinned to host CPUs that are not in the `isolcpus` reserved set.

+The vCPUs need to be given a real time CPU schedular policy. + +When configuring the `guest CPU count <../formatdomain.html#elementsCPUAllocation>`_, +do not include any CPU affinity are this stage:

+ +:: + + <vcpu placement='static'>4</vcpu> + +The guest CPUs now need to be placed individually. In this case, they will all +be put within the same host socket, such that they can be exposed as core +siblings. This is achieved using the `CPU tunning config <../formatdomain.html#elementsCPUTuning>`_: + +:: + + <cputune> + <emulatorpin cpuset="8-9"/> + <vcpupin vcpu="0" cpuset="12"/> + <vcpupin vcpu="1" cpuset="13"/> + <vcpupin vcpu="2" cpuset="14"/> + <vcpupin vcpu="3" cpuset="15"/> + <vcpusched vcpus='0-4' scheduler='fifo' priority='1'/> + </cputune> + +The `guest CPU model <formatdomain.html#elementsCPU>`_ now needs to be +configured to pass through the host model unchanged, with topology matching the +placement: + +:: + + <cpu mode='host-passthrough'> + <topology sockets='1' dies='1' cores='4' threads='1'/> + <feature policy='require' name='tsc-deadline'/> + </cpu> + +The performance monitoring unit virtualization needs to be disabled +via the `hypervisor features <../formatdomain.html#elementsFeatures>`_: + +:: + + <features> + ... + <pmu state='off'/> + </features> + + +Memory configuration +-------------------- + +The host memory used for guest RAM needs to be allocated from huge pages on the +second NUMA node, and all other memory allocation needs to be locked into RAM +with memory page sharing disabled. +This is achieved by using the `memory backing config <formatdomain.html#elementsMemoryBacking>`_: + +:: + + <memoryBacking> + <hugepages> + <page size="1" unit="G" nodeset="1"/> + </hugepages> + <locked/> + <nosharepages/> + </memoryBacking> + + +Device configuration +-------------------- + +Libvirt adds a few devices by default to maintain historical QEMU configuration +behaviour. It is unlikely these devices are required by real time guests, so it +is wise to disable them. Remove all USB controllers that may exist in the XML +config and replace them with: + +:: + + <controller type="usb" model="none"/> + +Similarly the memory balloon config should be changed to + +:: + + <memballoon model="none"/> + +If the guest had a graphical console at installation time this can also be +disabled, with remote access being over SSH, with a minimal serial console +for emergencies.