[libvirt] [PATCH 0/3] lxc: Implement emulator pin APIs to set/get cpuset
We can specify cpuset for a container defined with the xml like <vcpu placement='static' cpuset='0-3'> to achieve cpu isolation. It works when container is started. But there is no implements we can use to either change or get cpuset. The following patches implement the lxc driver methods for virDomainPinEmulator and virDomainGetEmulatorPinInfo. Also support container startup with emulator affinity info in xml. After these patches, we can set and get libvirt_lxc cpuset. Yue Wenyuan (3): lxc: Implement pin emulator for container startup lxc: Implement emulator pin API in lxc driver lxc: Implement geting emulator pin info API in lxc driver src/lxc/lxc_cgroup.c | 88 ++++++++++++++++++++ src/lxc/lxc_cgroup.h | 7 ++ src/lxc/lxc_controller.c | 4 + src/lxc/lxc_driver.c | 206 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 305 insertions(+) -- 1.7.12.4
From: Yue Wenyuan <yuewenyuan@huawei.com> This patch implements libvirt_lxc process pin with emulatorpin specified in xml. Signed-off-by: Wang Rui <moon.wangrui@huawei.com> Signed-off-by: Yue Wenyuan <yuewenyuan@huawei.com> --- src/lxc/lxc_cgroup.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++ src/lxc/lxc_cgroup.h | 4 +++ src/lxc/lxc_controller.c | 4 +++ 3 files changed, 76 insertions(+) diff --git a/src/lxc/lxc_cgroup.c b/src/lxc/lxc_cgroup.c index f9af31c..f696bf8 100644 --- a/src/lxc/lxc_cgroup.c +++ b/src/lxc/lxc_cgroup.c @@ -530,3 +530,71 @@ int virLXCCgroupSetup(virDomainDefPtr def, cleanup: return ret; } + +static int virLXCCgroupSetupCpusetTuneForEmulator(virDomainDefPtr def, + virCgroupPtr cgroup, + virBitmapPtr nodemask) +{ + int ret = -1; + char *mask = NULL; + + if (def->placement_mode != VIR_DOMAIN_CPU_PLACEMENT_MODE_AUTO) { + if (def->cputune.emulatorpin) { + if (!(mask = virBitmapFormat(def->cputune.emulatorpin->cpumask))) + return ret; + } else if (def->cpumask) { + if (!(mask = virBitmapFormat(def->cpumask))) + return ret; + } + if (virCgroupSetCpusetCpus(cgroup, mask) < 0) + goto cleanup; + } + + if (virDomainNumatuneMaybeFormatNodeset(def->numatune, nodemask, + &mask, -1) < 0) + goto cleanup; + + if (mask && virCgroupSetCpusetMems(cgroup, mask) < 0) + goto cleanup; + + ret = 0; + + cleanup: + VIR_FREE(mask); + return ret; +} + +int virLXCCgroupSetupForEmulator(virDomainDefPtr def, + virCgroupPtr cgroup, + virBitmapPtr nodemask) +{ + virCgroupPtr cgroup_emulator = NULL; + + if (!virCgroupHasController(cgroup, VIR_CGROUP_CONTROLLER_CPUSET)) + return 0; + + if (cgroup == NULL) + return 0; /* Not supported, so claim success */ + + if (virCgroupNewEmulator(cgroup, true, &cgroup_emulator) < 0) + goto error; + + if (virCgroupMoveTask(cgroup, cgroup_emulator) < 0) + goto error; + + if (virCgroupHasController(cgroup_emulator, VIR_CGROUP_CONTROLLER_CPUSET) && + virLXCCgroupSetupCpusetTuneForEmulator(def, cgroup_emulator, nodemask) < 0) + goto error; + + virCgroupFree(&cgroup_emulator); + return 0; + + error: + + if (cgroup_emulator) { + virCgroupRemove(cgroup_emulator); + virCgroupFree(&cgroup_emulator); + } + + return -1; +} diff --git a/src/lxc/lxc_cgroup.h b/src/lxc/lxc_cgroup.h index 0e78126..32086c5 100644 --- a/src/lxc/lxc_cgroup.h +++ b/src/lxc/lxc_cgroup.h @@ -33,6 +33,10 @@ int virLXCCgroupSetup(virDomainDefPtr def, virCgroupPtr cgroup, virBitmapPtr nodemask); +int virLXCCgroupSetupForEmulator(virDomainDefPtr def, + virCgroupPtr cgroup, + virBitmapPtr nodemask); + int virLXCCgroupGetMeminfo(virLXCMeminfoPtr meminfo); int diff --git a/src/lxc/lxc_controller.c b/src/lxc/lxc_controller.c index 1861dd6..1a62e20 100644 --- a/src/lxc/lxc_controller.c +++ b/src/lxc/lxc_controller.c @@ -698,6 +698,10 @@ static int virLXCControllerSetupResourceLimits(virLXCControllerPtr ctrl) if (virLXCCgroupSetup(ctrl->def, ctrl->cgroup, nodemask) < 0) goto cleanup; + VIR_DEBUG("Setting cgroup for lxc emulator"); + if (virLXCCgroupSetupForEmulator(ctrl->def, ctrl->cgroup, nodemask) < 0) + goto cleanup; + ret = 0; cleanup: virBitmapFree(nodemask); -- 1.7.12.4
On 09/04/2014 03:52 AM, Wang Rui wrote:
From: Yue Wenyuan <yuewenyuan@huawei.com>
This patch implements libvirt_lxc process pin with emulatorpin specified in xml.
Signed-off-by: Wang Rui <moon.wangrui@huawei.com> Signed-off-by: Yue Wenyuan <yuewenyuan@huawei.com> --- src/lxc/lxc_cgroup.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++ src/lxc/lxc_cgroup.h | 4 +++ src/lxc/lxc_controller.c | 4 +++ 3 files changed, 76 insertions(+)
I'm not an LXC expert, but I'll give this a go since it's been sitting around a while. I am curious why only the CPUSET (eg, <vcpuset ... cpuset="1-4,^3,6"> or <emulatorpin cpuset="1-3"/>) were handled and the <emulator_period> and <emulator_quota> were not?
diff --git a/src/lxc/lxc_cgroup.c b/src/lxc/lxc_cgroup.c index f9af31c..f696bf8 100644 --- a/src/lxc/lxc_cgroup.c +++ b/src/lxc/lxc_cgroup.c @@ -530,3 +530,71 @@ int virLXCCgroupSetup(virDomainDefPtr def, cleanup: return ret; } + +static int virLXCCgroupSetupCpusetTuneForEmulator(virDomainDefPtr def,
Change name virLXCCgroupSetupCpusetTuneForEmulator to virLXCSetupCgroupCpusetTunesForEmulator [1] note the (s) on Tune as well... Could have also gone with "CpusetCpusMems", but that really seemed too long
+ virCgroupPtr cgroup, + virBitmapPtr nodemask)
Although lxc_cgroup doesn't do it for the majority of the functions use: static int virLXCSetupCgroupEmulatorCpusetCpusMems(...) make sure to align arguments properly too.
+{ + int ret = -1; + char *mask = NULL; + + if (def->placement_mode != VIR_DOMAIN_CPU_PLACEMENT_MODE_AUTO) { + if (def->cputune.emulatorpin) { + if (!(mask = virBitmapFormat(def->cputune.emulatorpin->cpumask))) + return ret; + } else if (def->cpumask) { + if (!(mask = virBitmapFormat(def->cpumask))) + return ret; + }
At this point mask could still be NULL, thus you need a "if (mask && " prior to call...
+ if (virCgroupSetCpusetCpus(cgroup, mask) < 0) + goto cleanup; + }
And if MODE_AUTO, then does virLXCControllerSetupCpuAffinity called prior to this function satisfy the doc requirement to query numad? If so, then perhaps noting that as a comment prior to this hunk of code would be beneficial. If not, then that needs to be handled. It doesn't seem that way since I don't see comparable code in LXC to the qemuPrepareCpumap code that handles the host.numaCell. Also since you're reusing 'mask' below you'll need a VIR_FREE(mask); prior to the next set of calls; otherwise, you'll leak it.
+ + if (virDomainNumatuneMaybeFormatNodeset(def->numatune, nodemask, + &mask, -1) < 0) + goto cleanup; + + if (mask && virCgroupSetCpusetMems(cgroup, mask) < 0) + goto cleanup; + + ret = 0; + + cleanup: + VIR_FREE(mask); + return ret; +} + +int virLXCCgroupSetupForEmulator(virDomainDefPtr def, + virCgroupPtr cgroup, + virBitmapPtr nodemask)
The qemu function is qemuSetupCgroupForEmulator, thus this should be virLXCSetupCgroupForEmulator. [2] The function arguments need to be aligned and like above: int virLXCSetupCgroupForEmulator(...)
+{ + virCgroupPtr cgroup_emulator = NULL; + + if (!virCgroupHasController(cgroup, VIR_CGROUP_CONTROLLER_CPUSET)) + return 0; + + if (cgroup == NULL) + return 0; /* Not supported, so claim success */
This seems to be superfluous as the virCgroupHasController() will return false if (!cgroup), although I do see that it's a cut-n-paste of the similar qemu code.
+ + if (virCgroupNewEmulator(cgroup, true, &cgroup_emulator) < 0) + goto error; + + if (virCgroupMoveTask(cgroup, cgroup_emulator) < 0) + goto error; + + if (virCgroupHasController(cgroup_emulator, VIR_CGROUP_CONTROLLER_CPUSET) && ^^^^^^^^^^^^^^^ qemu code uses priv->cgroup, so should this code use 'cgroup' instead? or is the qemu code incorrect and should be fixed?
+ virLXCCgroupSetupCpusetTuneForEmulator(def, cgroup_emulator, nodemask) < 0)
[1] Use new name...
+ goto error; + + virCgroupFree(&cgroup_emulator); + return 0; + + error: + + if (cgroup_emulator) { + virCgroupRemove(cgroup_emulator); + virCgroupFree(&cgroup_emulator); + } + + return -1; +} diff --git a/src/lxc/lxc_cgroup.h b/src/lxc/lxc_cgroup.h index 0e78126..32086c5 100644 --- a/src/lxc/lxc_cgroup.h +++ b/src/lxc/lxc_cgroup.h @@ -33,6 +33,10 @@ int virLXCCgroupSetup(virDomainDefPtr def, virCgroupPtr cgroup, virBitmapPtr nodemask);
+int virLXCCgroupSetupForEmulator(virDomainDefPtr def, + virCgroupPtr cgroup, + virBitmapPtr nodemask); + [2] See note above about name change s/CgroupSetup/SetupCgroup
Also the argument alignment is off - yes I see the previous function prototype is too, but lets at least do this one right and fix the other one later in either a follow up or separate patch prior to this patch.
int virLXCCgroupGetMeminfo(virLXCMeminfoPtr meminfo);
int diff --git a/src/lxc/lxc_controller.c b/src/lxc/lxc_controller.c index 1861dd6..1a62e20 100644 --- a/src/lxc/lxc_controller.c +++ b/src/lxc/lxc_controller.c @@ -698,6 +698,10 @@ static int virLXCControllerSetupResourceLimits(virLXCControllerPtr ctrl) if (virLXCCgroupSetup(ctrl->def, ctrl->cgroup, nodemask) < 0) goto cleanup;
+ VIR_DEBUG("Setting cgroup for lxc emulator"); + if (virLXCCgroupSetupForEmulator(ctrl->def, ctrl->cgroup, nodemask) < 0)
[2] See note above about name change s/CgroupSetup/SetupCgroup
+ goto cleanup; + ret = 0; cleanup: virBitmapFree(nodemask);
You need to update docs/formatdomain.html.in as well to add the LXC Since 1.2.x (whenever this gets in) to various bits and parts of the <cputune> and <vcpus> descriptions. John
On 2014/10/22 23:34, John Ferlan wrote:
On 09/04/2014 03:52 AM, Wang Rui wrote:
From: Yue Wenyuan <yuewenyuan@huawei.com>
This patch implements libvirt_lxc process pin with emulatorpin specified in xml.
Signed-off-by: Wang Rui <moon.wangrui@huawei.com> Signed-off-by: Yue Wenyuan <yuewenyuan@huawei.com> --- src/lxc/lxc_cgroup.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++ src/lxc/lxc_cgroup.h | 4 +++ src/lxc/lxc_controller.c | 4 +++ 3 files changed, 76 insertions(+)
I'm not an LXC expert, but I'll give this a go since it's been sitting around a while.
I am curious why only the CPUSET (eg, <vcpuset ... cpuset="1-4,^3,6"> or <emulatorpin cpuset="1-3"/>) were handled and the <emulator_period> and <emulator_quota> were not?
Hi, John. Thank you for your review. These patches are only the implementation of emulator pin. It is ture that <emulator_period> and <emulator_quota> are not supported by lxc. I think It can be handled in the later patch if these patches are ACKed. [...]
+{ + int ret = -1; + char *mask = NULL; + + if (def->placement_mode != VIR_DOMAIN_CPU_PLACEMENT_MODE_AUTO) { + if (def->cputune.emulatorpin) { + if (!(mask = virBitmapFormat(def->cputune.emulatorpin->cpumask))) + return ret; + } else if (def->cpumask) { + if (!(mask = virBitmapFormat(def->cpumask))) + return ret; + }
At this point mask could still be NULL, thus you need a "if (mask && " prior to call...
+ if (virCgroupSetCpusetCpus(cgroup, mask) < 0) + goto cleanup; + }
And if MODE_AUTO, then does virLXCControllerSetupCpuAffinity called prior to this function satisfy the doc requirement to query numad? If so, then perhaps noting that as a comment prior to this hunk of code would be beneficial. If not, then that needs to be handled. It doesn't seem that way since I don't see comparable code in LXC to the qemuPrepareCpumap code that handles the host.numaCell.
Thanks to point that. I think virLXCControllerGetNumadAdvice which is called in virLXCControllerSetupResourceLimits is the comparable function to qemuPrepareCpumap. Maybe it's another bug.
Also since you're reusing 'mask' below you'll need a VIR_FREE(mask); prior to the next set of calls; otherwise, you'll leak it.
+ + if (virDomainNumatuneMaybeFormatNodeset(def->numatune, nodemask, + &mask, -1) < 0) + goto cleanup; + + if (mask && virCgroupSetCpusetMems(cgroup, mask) < 0) + goto cleanup; + + ret = 0; + + cleanup: + VIR_FREE(mask); + return ret; +} + +int virLXCCgroupSetupForEmulator(virDomainDefPtr def, + virCgroupPtr cgroup, + virBitmapPtr nodemask)
[...]
Also the argument alignment is off - yes I see the previous function prototype is too, but lets at least do this one right and fix the other one later in either a follow up or separate patch prior to this patch.
Some code is not well aligned. I can fix that as your suggestion. I'll send patches V2 after release 1.2.10.
From: Yue Wenyuan <yuewenyuan@huawei.com> Implement the lxc driver method for virDomainPinEmulator to set container's cpuset. Signed-off-by: Wang Rui <moon.wangrui@huawei.com> Signed-off-by: Yue Wenyuan <yuewenyuan@huawei.com> --- src/lxc/lxc_cgroup.c | 20 ++++++++ src/lxc/lxc_cgroup.h | 3 ++ src/lxc/lxc_driver.c | 131 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 154 insertions(+) diff --git a/src/lxc/lxc_cgroup.c b/src/lxc/lxc_cgroup.c index f696bf8..7dc7c9b 100644 --- a/src/lxc/lxc_cgroup.c +++ b/src/lxc/lxc_cgroup.c @@ -531,6 +531,26 @@ int virLXCCgroupSetup(virDomainDefPtr def, return ret; } +int +virLXCSetupCgroupEmulatorPin(virCgroupPtr cgroup, + virBitmapPtr cpumask) +{ + int ret = -1; + char *new_cpus = NULL; + + if (!(new_cpus = virBitmapFormat(cpumask))) + goto cleanup; + + if (virCgroupSetCpusetCpus(cgroup, new_cpus) < 0) + goto cleanup; + + ret = 0; + + cleanup: + VIR_FREE(new_cpus); + return ret; +} + static int virLXCCgroupSetupCpusetTuneForEmulator(virDomainDefPtr def, virCgroupPtr cgroup, virBitmapPtr nodemask) diff --git a/src/lxc/lxc_cgroup.h b/src/lxc/lxc_cgroup.h index 32086c5..8011a32 100644 --- a/src/lxc/lxc_cgroup.h +++ b/src/lxc/lxc_cgroup.h @@ -33,6 +33,9 @@ int virLXCCgroupSetup(virDomainDefPtr def, virCgroupPtr cgroup, virBitmapPtr nodemask); +int virLXCSetupCgroupEmulatorPin(virCgroupPtr cgroup, + virBitmapPtr cpumask); + int virLXCCgroupSetupForEmulator(virDomainDefPtr def, virCgroupPtr cgroup, virBitmapPtr nodemask); diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c index 66d708a..9e3a877 100644 --- a/src/lxc/lxc_driver.c +++ b/src/lxc/lxc_driver.c @@ -1264,6 +1264,136 @@ lxcDomainCreateXML(virConnectPtr conn, return lxcDomainCreateXMLWithFiles(conn, xml, 0, NULL, flags); } +static int +lxcDomainPinEmulator(virDomainPtr dom, + unsigned char *cpumap, + int maplen, + unsigned int flags) +{ + virLXCDriverPtr driver = dom->conn->privateData; + virDomainObjPtr vm; + virCgroupPtr cgroup_emulator = NULL; + pid_t pid; + virDomainDefPtr persistentDef = NULL; + int ret = -1; + virLXCDomainObjPrivatePtr priv; + bool doReset = false; + virBitmapPtr pcpumap = NULL; + virLXCDriverConfigPtr cfg = NULL; + virCapsPtr caps = NULL; + + virCheckFlags(VIR_DOMAIN_AFFECT_LIVE | + VIR_DOMAIN_AFFECT_CONFIG, -1); + + cfg = virLXCDriverGetConfig(driver); + + if (!(vm = lxcDomObjFromDomain(dom))) + goto cleanup; + + if (virDomainPinEmulatorEnsureACL(dom->conn, vm->def, flags) < 0) + goto cleanup; + + if (!(caps = virLXCDriverGetCapabilities(driver, false))) + goto cleanup; + + if (vm->def->placement_mode == VIR_DOMAIN_CPU_PLACEMENT_MODE_AUTO) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("Changing affinity for emulator thread dynamically " + "is not allowed when CPU placement is 'auto'")); + goto cleanup; + } + + if (virDomainLiveConfigHelperMethod(caps, driver->xmlopt, vm, &flags, + &persistentDef) < 0) + goto cleanup; + + priv = vm->privateData; + + pcpumap = virBitmapNewData(cpumap, maplen); + if (!pcpumap) + goto cleanup; + + if (virBitmapIsAllClear(pcpumap)) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Empty cpu list for pinning")); + goto cleanup; + } + + if (virBitmapIsAllSet(pcpumap)) + doReset = true; + + pid = vm->pid; + + if (flags & VIR_DOMAIN_AFFECT_LIVE) { + if (virCgroupHasController(priv->cgroup, + VIR_CGROUP_CONTROLLER_CPUSET)) { + if (virCgroupNewEmulator(priv->cgroup, false, &cgroup_emulator) < 0) + goto cleanup; + if (virLXCSetupCgroupEmulatorPin(cgroup_emulator, + pcpumap) < 0) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("failed to set cpuset.cpus in cgroup " + "for lxc emulator")); + goto cleanup; + } + } else { + virReportError(VIR_ERR_SYSTEM_ERROR, "%s", + _("there is not cgroup controller for " + "lxc emulator")); + goto cleanup; + } + if (doReset) { + if (virDomainEmulatorPinDel(vm->def) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to delete emulatorpin xml of " + "a running domain")); + goto cleanup; + } + } else { + if (virDomainEmulatorPinAdd(vm->def, cpumap, maplen) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to update or add emulatorpin xml " + "of a running domain")); + goto cleanup; + } + } + if (virDomainSaveStatus(driver->xmlopt, cfg->stateDir, vm) < 0) + goto cleanup; + } + + if (flags & VIR_DOMAIN_AFFECT_CONFIG) { + if (doReset) { + if (virDomainEmulatorPinDel(persistentDef) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to delete emulatorpin xml of " + "a persistent domain")); + goto cleanup; + } + } else { + if (virDomainEmulatorPinAdd(persistentDef, cpumap, maplen) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to update or add emulatorpin xml " + "of a persistent domain")); + goto cleanup; + } + } + + ret = virDomainSaveConfig(cfg->configDir, persistentDef); + goto cleanup; + } + + ret = 0; + + cleanup: + if (cgroup_emulator) + virCgroupFree(&cgroup_emulator); + virBitmapFree(pcpumap); + virObjectUnref(caps); + if (vm) + virObjectUnlock(vm); + virObjectUnref(cfg); + return ret; +} static int lxcDomainGetSecurityLabel(virDomainPtr dom, virSecurityLabelPtr seclabel) { @@ -5700,6 +5830,7 @@ static virDriver lxcDriver = { .connectListAllDomains = lxcConnectListAllDomains, /* 0.9.13 */ .domainCreateXML = lxcDomainCreateXML, /* 0.4.4 */ .domainCreateXMLWithFiles = lxcDomainCreateXMLWithFiles, /* 1.1.1 */ + .domainPinEmulator = lxcDomainPinEmulator, /*1.2.9*/ .domainLookupByID = lxcDomainLookupByID, /* 0.4.2 */ .domainLookupByUUID = lxcDomainLookupByUUID, /* 0.4.2 */ .domainLookupByName = lxcDomainLookupByName, /* 0.4.2 */ -- 1.7.12.4
On 09/04/2014 03:52 AM, Wang Rui wrote:
From: Yue Wenyuan <yuewenyuan@huawei.com>
Implement the lxc driver method for virDomainPinEmulator to set container's cpuset.
Signed-off-by: Wang Rui <moon.wangrui@huawei.com> Signed-off-by: Yue Wenyuan <yuewenyuan@huawei.com> --- src/lxc/lxc_cgroup.c | 20 ++++++++ src/lxc/lxc_cgroup.h | 3 ++ src/lxc/lxc_driver.c | 131 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 154 insertions(+)
diff --git a/src/lxc/lxc_cgroup.c b/src/lxc/lxc_cgroup.c index f696bf8..7dc7c9b 100644 --- a/src/lxc/lxc_cgroup.c +++ b/src/lxc/lxc_cgroup.c @@ -531,6 +531,26 @@ int virLXCCgroupSetup(virDomainDefPtr def, return ret; }
+int +virLXCSetupCgroupEmulatorPin(virCgroupPtr cgroup, + virBitmapPtr cpumask) +{ + int ret = -1; + char *new_cpus = NULL; + + if (!(new_cpus = virBitmapFormat(cpumask))) + goto cleanup; + + if (virCgroupSetCpusetCpus(cgroup, new_cpus) < 0) + goto cleanup; + + ret = 0; + + cleanup: + VIR_FREE(new_cpus); + return ret; +} + static int virLXCCgroupSetupCpusetTuneForEmulator(virDomainDefPtr def, virCgroupPtr cgroup, virBitmapPtr nodemask) diff --git a/src/lxc/lxc_cgroup.h b/src/lxc/lxc_cgroup.h index 32086c5..8011a32 100644 --- a/src/lxc/lxc_cgroup.h +++ b/src/lxc/lxc_cgroup.h @@ -33,6 +33,9 @@ int virLXCCgroupSetup(virDomainDefPtr def, virCgroupPtr cgroup, virBitmapPtr nodemask);
+int virLXCSetupCgroupEmulatorPin(virCgroupPtr cgroup, + virBitmapPtr cpumask); +
alignment issues with the arguments
int virLXCCgroupSetupForEmulator(virDomainDefPtr def, virCgroupPtr cgroup, virBitmapPtr nodemask); diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c index 66d708a..9e3a877 100644 --- a/src/lxc/lxc_driver.c +++ b/src/lxc/lxc_driver.c @@ -1264,6 +1264,136 @@ lxcDomainCreateXML(virConnectPtr conn, return lxcDomainCreateXMLWithFiles(conn, xml, 0, NULL, flags); }
+static int +lxcDomainPinEmulator(virDomainPtr dom, + unsigned char *cpumap, + int maplen, + unsigned int flags)
Alignment issues with arguments.
+{ + virLXCDriverPtr driver = dom->conn->privateData; + virDomainObjPtr vm; + virCgroupPtr cgroup_emulator = NULL; + pid_t pid; + virDomainDefPtr persistentDef = NULL; + int ret = -1; + virLXCDomainObjPrivatePtr priv; + bool doReset = false; + virBitmapPtr pcpumap = NULL; + virLXCDriverConfigPtr cfg = NULL; + virCapsPtr caps = NULL; + + virCheckFlags(VIR_DOMAIN_AFFECT_LIVE | + VIR_DOMAIN_AFFECT_CONFIG, -1); + + cfg = virLXCDriverGetConfig(driver); + + if (!(vm = lxcDomObjFromDomain(dom))) + goto cleanup; + + if (virDomainPinEmulatorEnsureACL(dom->conn, vm->def, flags) < 0) + goto cleanup; + + if (!(caps = virLXCDriverGetCapabilities(driver, false))) + goto cleanup; + + if (vm->def->placement_mode == VIR_DOMAIN_CPU_PLACEMENT_MODE_AUTO) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("Changing affinity for emulator thread dynamically " + "is not allowed when CPU placement is 'auto'")); + goto cleanup; + } + + if (virDomainLiveConfigHelperMethod(caps, driver->xmlopt, vm, &flags, + &persistentDef) < 0) + goto cleanup; + + priv = vm->privateData; + + pcpumap = virBitmapNewData(cpumap, maplen); + if (!pcpumap) + goto cleanup; + + if (virBitmapIsAllClear(pcpumap)) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Empty cpu list for pinning")); + goto cleanup; + } + + if (virBitmapIsAllSet(pcpumap)) + doReset = true; + + pid = vm->pid;
This is set, but never used - compilation failure on my Fedora system
+ + if (flags & VIR_DOMAIN_AFFECT_LIVE) { + if (virCgroupHasController(priv->cgroup, + VIR_CGROUP_CONTROLLER_CPUSET)) {
Alignment issues
+ if (virCgroupNewEmulator(priv->cgroup, false, &cgroup_emulator) < 0) + goto cleanup; + if (virLXCSetupCgroupEmulatorPin(cgroup_emulator, + pcpumap) < 0) {
Alignment issues
+ virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("failed to set cpuset.cpus in cgroup " + "for lxc emulator")); + goto cleanup; + } + } else {
And here is the missing line from the comparable qemu code, e.g. a call to virProcessSetAffinity(pid, pcpumap). I note that the LXC code during virLXCControllerSetupCpuAffinity will call SetAffinity with 0 (zero) as the pid. Whether that's relevant here or not I'm not quite sure.
+ virReportError(VIR_ERR_SYSTEM_ERROR, "%s", + _("there is not cgroup controller for " + "lxc emulator"));
Perhaps better said "no cpuset cgroup controller found for lxc emulator" - although just because HasController/CPUSET fails, we won't continue which doesn't seem right.
+ goto cleanup; + } + if (doReset) { + if (virDomainEmulatorPinDel(vm->def) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to delete emulatorpin xml of " + "a running domain")); + goto cleanup; + } + } else { + if (virDomainEmulatorPinAdd(vm->def, cpumap, maplen) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to update or add emulatorpin xml " + "of a running domain")); + goto cleanup; + } + } + if (virDomainSaveStatus(driver->xmlopt, cfg->stateDir, vm) < 0) + goto cleanup;
The qemu code will issue a (new) event here for VIR_DOMAIN_TUNABLE_CPU_EMULATORPIN... Is that applicable here? If so and you add it, don't forget to clean up the event during cleanup:
+ } + + if (flags & VIR_DOMAIN_AFFECT_CONFIG) { + if (doReset) { + if (virDomainEmulatorPinDel(persistentDef) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to delete emulatorpin xml of " + "a persistent domain")); + goto cleanup; + } + } else { + if (virDomainEmulatorPinAdd(persistentDef, cpumap, maplen) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to update or add emulatorpin xml " + "of a persistent domain")); + goto cleanup; + } + } + + ret = virDomainSaveConfig(cfg->configDir, persistentDef); + goto cleanup; + } + + ret = 0; + + cleanup: + if (cgroup_emulator) + virCgroupFree(&cgroup_emulator); + virBitmapFree(pcpumap); + virObjectUnref(caps); + if (vm) + virObjectUnlock(vm); + virObjectUnref(cfg); + return ret; +}
static int lxcDomainGetSecurityLabel(virDomainPtr dom, virSecurityLabelPtr seclabel) { @@ -5700,6 +5830,7 @@ static virDriver lxcDriver = { .connectListAllDomains = lxcConnectListAllDomains, /* 0.9.13 */ .domainCreateXML = lxcDomainCreateXML, /* 0.4.4 */ .domainCreateXMLWithFiles = lxcDomainCreateXMLWithFiles, /* 1.1.1 */ + .domainPinEmulator = lxcDomainPinEmulator, /*1.2.9*/
Comment spacing is off, will now be at least /* 1.2.10 */ John
.domainLookupByID = lxcDomainLookupByID, /* 0.4.2 */ .domainLookupByUUID = lxcDomainLookupByUUID, /* 0.4.2 */ .domainLookupByName = lxcDomainLookupByName, /* 0.4.2 */
From: Yue Wenyuan <yuewenyuan@huawei.com> Implement the lxc driver method for virDomainGetEmulatorPinInfo to get container's cpuset. Signed-off-by: Wang Rui <moon.wangrui@huawei.com> Signed-off-by: Yue Wenyuan <yuewenyuan@huawei.com> --- src/lxc/lxc_driver.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c index 9e3a877..6694cf3 100644 --- a/src/lxc/lxc_driver.c +++ b/src/lxc/lxc_driver.c @@ -1395,6 +1395,80 @@ lxcDomainPinEmulator(virDomainPtr dom, return ret; } +static int +lxcDomainGetEmulatorPinInfo(virDomainPtr dom, + unsigned char *cpumaps, + int maplen, + unsigned int flags) +{ + virLXCDriverPtr driver = dom->conn->privateData; + virDomainObjPtr vm = NULL; + virDomainDefPtr targetDef = NULL; + int ret = -1; + int maxcpu, hostcpus, pcpu; + virBitmapPtr cpumask = NULL; + bool pinned; + virCapsPtr caps = NULL; + + virCheckFlags(VIR_DOMAIN_AFFECT_LIVE | + VIR_DOMAIN_AFFECT_CONFIG, -1); + + if (!(vm = lxcDomObjFromDomain(dom))) + goto cleanup; + + if (virDomainGetEmulatorPinInfoEnsureACL(dom->conn, vm->def) < 0) + goto cleanup; + + if (!(caps = virLXCDriverGetCapabilities(driver, false))) + goto cleanup; + + if (virDomainLiveConfigHelperMethod(caps, driver->xmlopt, + vm, &flags, &targetDef) < 0) + goto cleanup; + + if (flags & VIR_DOMAIN_AFFECT_LIVE) + targetDef = vm->def; + + sa_assert(targetDef); + + if ((hostcpus = nodeGetCPUCount()) < 0) + goto cleanup; + + maxcpu = maplen * 8; + if (maxcpu > hostcpus) + maxcpu = hostcpus; + + /* initialize cpumaps */ + memset(cpumaps, 0xff, maplen); + if (maxcpu % 8) { + cpumaps[maplen - 1] &= (1 << maxcpu % 8) - 1; + } + + if (targetDef->cputune.emulatorpin) { + cpumask = targetDef->cputune.emulatorpin->cpumask; + } else if (targetDef->cpumask) { + cpumask = targetDef->cpumask; + } else { + ret = 0; + goto cleanup; + } + + for (pcpu = 0; pcpu < maxcpu; pcpu++) { + if (virBitmapGetBit(cpumask, pcpu, &pinned) < 0) + goto cleanup; + if (!pinned) + VIR_UNUSE_CPU(cpumaps, pcpu); + } + + ret = 1; + + cleanup: + if (vm) + virObjectUnlock(vm); + virObjectUnref(caps); + return ret; +} + static int lxcDomainGetSecurityLabel(virDomainPtr dom, virSecurityLabelPtr seclabel) { virLXCDriverPtr driver = dom->conn->privateData; @@ -5831,6 +5905,7 @@ static virDriver lxcDriver = { .domainCreateXML = lxcDomainCreateXML, /* 0.4.4 */ .domainCreateXMLWithFiles = lxcDomainCreateXMLWithFiles, /* 1.1.1 */ .domainPinEmulator = lxcDomainPinEmulator, /*1.2.9*/ + .domainGetEmulatorPinInfo = lxcDomainGetEmulatorPinInfo, /*1.2.9*/ .domainLookupByID = lxcDomainLookupByID, /* 0.4.2 */ .domainLookupByUUID = lxcDomainLookupByUUID, /* 0.4.2 */ .domainLookupByName = lxcDomainLookupByName, /* 0.4.2 */ -- 1.7.12.4
Fix the $SUBJ "s/geting/getting" On 09/04/2014 03:52 AM, Wang Rui wrote:
From: Yue Wenyuan <yuewenyuan@huawei.com>
Implement the lxc driver method for virDomainGetEmulatorPinInfo to get container's cpuset.
Signed-off-by: Wang Rui <moon.wangrui@huawei.com> Signed-off-by: Yue Wenyuan <yuewenyuan@huawei.com> --- src/lxc/lxc_driver.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)
diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c index 9e3a877..6694cf3 100644 --- a/src/lxc/lxc_driver.c +++ b/src/lxc/lxc_driver.c @@ -1395,6 +1395,80 @@ lxcDomainPinEmulator(virDomainPtr dom, return ret; }
Since the following is essentially a copy of the qemu function with the only changes being "LXC" for "QEMU" or "lxc" for "qemu" - it seems that it should be possible to create some sort of common vir/util function especially for all the lines between [1] and [2] below. Of course that's probably true in other areas, but since I was comparing here I guess it
+static int +lxcDomainGetEmulatorPinInfo(virDomainPtr dom, + unsigned char *cpumaps, + int maplen, + unsigned int flags) +{ + virLXCDriverPtr driver = dom->conn->privateData; + virDomainObjPtr vm = NULL; + virDomainDefPtr targetDef = NULL; + int ret = -1; + int maxcpu, hostcpus, pcpu; + virBitmapPtr cpumask = NULL; + bool pinned; + virCapsPtr caps = NULL; + + virCheckFlags(VIR_DOMAIN_AFFECT_LIVE | + VIR_DOMAIN_AFFECT_CONFIG, -1); + + if (!(vm = lxcDomObjFromDomain(dom))) + goto cleanup; + + if (virDomainGetEmulatorPinInfoEnsureACL(dom->conn, vm->def) < 0) + goto cleanup; + + if (!(caps = virLXCDriverGetCapabilities(driver, false))) + goto cleanup; + + if (virDomainLiveConfigHelperMethod(caps, driver->xmlopt, + vm, &flags, &targetDef) < 0) + goto cleanup; +
[1]
+ if (flags & VIR_DOMAIN_AFFECT_LIVE) + targetDef = vm->def; + + sa_assert(targetDef); + + if ((hostcpus = nodeGetCPUCount()) < 0) + goto cleanup; + + maxcpu = maplen * 8; + if (maxcpu > hostcpus) + maxcpu = hostcpus; + + /* initialize cpumaps */ + memset(cpumaps, 0xff, maplen); + if (maxcpu % 8) { + cpumaps[maplen - 1] &= (1 << maxcpu % 8) - 1; + } + + if (targetDef->cputune.emulatorpin) { + cpumask = targetDef->cputune.emulatorpin->cpumask; + } else if (targetDef->cpumask) { + cpumask = targetDef->cpumask; + } else { + ret = 0; + goto cleanup; + } + + for (pcpu = 0; pcpu < maxcpu; pcpu++) { + if (virBitmapGetBit(cpumask, pcpu, &pinned) < 0) + goto cleanup; + if (!pinned) + VIR_UNUSE_CPU(cpumaps, pcpu); + }
[2]
+ + ret = 1; + + cleanup: + if (vm) + virObjectUnlock(vm); + virObjectUnref(caps); + return ret; +} + static int lxcDomainGetSecurityLabel(virDomainPtr dom, virSecurityLabelPtr seclabel) { virLXCDriverPtr driver = dom->conn->privateData; @@ -5831,6 +5905,7 @@ static virDriver lxcDriver = { .domainCreateXML = lxcDomainCreateXML, /* 0.4.4 */ .domainCreateXMLWithFiles = lxcDomainCreateXMLWithFiles, /* 1.1.1 */ .domainPinEmulator = lxcDomainPinEmulator, /*1.2.9*/ + .domainGetEmulatorPinInfo = lxcDomainGetEmulatorPinInfo, /*1.2.9*/
Comment spacing is off, will now be at least /* 1.2.10 */ John
.domainLookupByID = lxcDomainLookupByID, /* 0.4.2 */ .domainLookupByUUID = lxcDomainLookupByUUID, /* 0.4.2 */ .domainLookupByName = lxcDomainLookupByName, /* 0.4.2 */
Ping? On 2014/9/4 15:52, Wang Rui wrote:
We can specify cpuset for a container defined with the xml like <vcpu placement='static' cpuset='0-3'> to achieve cpu isolation. It works when container is started. But there is no implements we can use to either change or get cpuset.
The following patches implement the lxc driver methods for virDomainPinEmulator and virDomainGetEmulatorPinInfo. Also support container startup with emulator affinity info in xml.
After these patches, we can set and get libvirt_lxc cpuset.
Yue Wenyuan (3): lxc: Implement pin emulator for container startup lxc: Implement emulator pin API in lxc driver lxc: Implement geting emulator pin info API in lxc driver
src/lxc/lxc_cgroup.c | 88 ++++++++++++++++++++ src/lxc/lxc_cgroup.h | 7 ++ src/lxc/lxc_controller.c | 4 + src/lxc/lxc_driver.c | 206 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 305 insertions(+)
Ping? I found some wrong indentation in the 3 patches.
On Fri, 2014-10-10 at 13:58 +0800, Wang Rui wrote: like following in patch 1/3: +static int virLXCCgroupSetupCpusetTuneForEmulator(virDomainDefPtr def, + virCgroupPtr cgroup, + virBitmapPtr nodemask) Thanks, Chen
On 2014/9/4 15:52, Wang Rui wrote:
We can specify cpuset for a container defined with the xml like <vcpu placement='static' cpuset='0-3'> to achieve cpu isolation. It works when container is started. But there is no implements we can use to either change or get cpuset.
The following patches implement the lxc driver methods for virDomainPinEmulator and virDomainGetEmulatorPinInfo. Also support container startup with emulator affinity info in xml.
After these patches, we can set and get libvirt_lxc cpuset.
Yue Wenyuan (3): lxc: Implement pin emulator for container startup lxc: Implement emulator pin API in lxc driver lxc: Implement geting emulator pin info API in lxc driver
src/lxc/lxc_cgroup.c | 88 ++++++++++++++++++++ src/lxc/lxc_cgroup.h | 7 ++ src/lxc/lxc_controller.c | 4 + src/lxc/lxc_driver.c | 206 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 305 insertions(+)
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
participants (3)
-
Chen, Fan -
John Ferlan -
Wang Rui