Re: [libvirt] [RFC] Memory controller exploitation in libvirt

Subject: [RFC] Memory controller exploitation in libvirt Memory CGroup is a kernel feature that can be exploited effectively in the current libvirt/qemu driver. Here is a shot at that. At present, QEmu uses memory ballooning feature, where the memory can be inflated/deflated as and when needed, co-operatively between the host and the guest. There should be some mechanism where the host can have more control over the guests memory usage. Memory CGroup provides features such as hard-limit and soft-limit for memory, and hard-limit for swap area. Design 1: Provide new API and XML changes for resource management ================================================================= All the memory controller tunables are not supported with the current abstractions provided by the libvirt API. libvirt works on various OS. This new API will support GNU/Linux initially and as and when other platforms starts supporting memory tunables, the interface could be enabled for them. Adding following two function pointer to the virDriver interface. 1) domainSetMemoryParameters: which would take one or more name-value pairs. This makes the API extensible, and agnostic to the kind of parameters supported by various Hypervisors. 2) domainGetMemoryParameters: For getting current memory parameters Corresponding libvirt public API: int virDomainSetMemoryParamters (virDomainPtr domain, virMemoryParamterPtr params, unsigned int nparams); int virDomainGetMemoryParamters (virDomainPtr domain, virMemoryParamterPtr params, unsigned int nparams);

Parameter list supported: MemoryHardLimits (memory.limits_in_bytes) - Maximum memory MemorySoftLimits (memory.softlimit_in_bytes) - Desired memory

MemoryMinimumGaurantee - Minimum memory required (without this amount of memory, VM should not be started) SwapHardLimits (memory.memsw_limit_in_bytes) - Maximum swap SwapSoftLimits (Currently not supported by kernel) - Desired swap space

Tunables memory.limit_in_bytes, memory.softlimit_in_bytes and memory.memsw_limit_in_bytes are provided by the memory controller in the Linux kernel. I am not an expert here, so just listing what new elements need to be added to the XML schema: <define name="resource"> <element memory> <element memoryHardLimit/> <element memorySoftLimit/> <element memoryMinGaurantee/> <element swapHardLimit/> <element swapSoftLimit/> </element> </define>

Pros: * Support all the tunables exported by the kernel * More tunables can be added as and when required Cons: * Code changes would touch various levels * Might need to redefine(changing the scope) of existing memory API. Currently, domainSetMemory is used to set limit_in_bytes in LXC and memory ballooning in QEmu. While the domainSetMaxMemory is not defined in QEmu and in case of LXC it is setting the internal object's maxmem variable. Future: * Later on, CPU/IO/Network controllers related tunables can be added/enhanced along with the APIs/XML elements: CPUHardLimit CPUSoftLimit CPUShare CPUPercentage IO_BW_Softlimit IO_BW_Hardlimit IO_BW_percentage * libvirt-cim support for resource management Design 2: Reuse the current memory APIs in libvirt ================================================== Use memory.limit_in_bytes to tweak memory hard limits Init - Set the memory.limit_in_bytes to maximum mem. Claiming memory from guest: a) Reduce balloon size b) If the guest does not co-operate(How do we know?), reduce memory.limit_in_bytes.

Allocating memory more than max memory: How to solve this? As we have already set the max balloon size. We can only play within this! Pros: * Few changes * Is not intrusive Cons: * SetMemory and SetMaxMemory usage is confusing. * SetMemory is too generic a name, it does not cover all the tunables. * Does not support memory softlimit * Does not have support to reserve the memory swap region * This solution is not extensible IMO, "Design 1" is more generic and extensible for various memory tuneables.

Nikunj