2:52 a.m.
We can specify cpuset for a container defined with the xml
like <vcpu placement='static' cpuset='0-3'> to achieve cpu
isolation. It works when container is started. But there
is no implements we can use to either change or get cpuset.
The following patches implement the lxc driver methods for
virDomainPinEmulator and virDomainGetEmulatorPinInfo. Also
support container startup with emulator affinity info in xml.
After these patches, we can set and get libvirt_lxc cpuset.
Yue Wenyuan (3):
lxc: Implement pin emulator for container startup
lxc: Implement emulator pin API in lxc driver
lxc: Implement geting emulator pin info API in lxc driver
src/lxc/lxc_cgroup.c | 88 ++++++++++++++++++++
src/lxc/lxc_cgroup.h | 7 ++
src/lxc/lxc_controller.c | 4 +
src/lxc/lxc_driver.c | 206 +++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 305 insertions(+)
--
1.7.12.4
2:52 a.m.
New subject: [libvirt] [PATCH 1/3] lxc: Implement pin emulator for container startup
From: Yue Wenyuan <yuewenyuan(a)huawei.com>
This patch implements libvirt_lxc process pin with emulatorpin
specified in xml.
Signed-off-by: Wang Rui <moon.wangrui(a)huawei.com>
Signed-off-by: Yue Wenyuan <yuewenyuan(a)huawei.com>
---
src/lxc/lxc_cgroup.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++
src/lxc/lxc_cgroup.h | 4 +++
src/lxc/lxc_controller.c | 4 +++
3 files changed, 76 insertions(+)
diff --git a/src/lxc/lxc_cgroup.c b/src/lxc/lxc_cgroup.c
index f9af31c..f696bf8 100644
--- a/src/lxc/lxc_cgroup.c
+++ b/src/lxc/lxc_cgroup.c
@@ -530,3 +530,71 @@ int virLXCCgroupSetup(virDomainDefPtr def,
cleanup:
return ret;
}
+
+static int virLXCCgroupSetupCpusetTuneForEmulator(virDomainDefPtr def,
+ virCgroupPtr cgroup,
+ virBitmapPtr nodemask)
+{
+ int ret = -1;
+ char *mask = NULL;
+
+ if (def->placement_mode != VIR_DOMAIN_CPU_PLACEMENT_MODE_AUTO) {
+ if (def->cputune.emulatorpin) {
+ if (!(mask = virBitmapFormat(def->cputune.emulatorpin->cpumask)))
+ return ret;
+ } else if (def->cpumask) {
+ if (!(mask = virBitmapFormat(def->cpumask)))
+ return ret;
+ }
+ if (virCgroupSetCpusetCpus(cgroup, mask) < 0)
+ goto cleanup;
+ }
+
+ if (virDomainNumatuneMaybeFormatNodeset(def->numatune, nodemask,
+ &mask, -1) < 0)
+ goto cleanup;
+
+ if (mask && virCgroupSetCpusetMems(cgroup, mask) < 0)
+ goto cleanup;
+
+ ret = 0;
+
+ cleanup:
+ VIR_FREE(mask);
+ return ret;
+}
+
+int virLXCCgroupSetupForEmulator(virDomainDefPtr def,
+ virCgroupPtr cgroup,
+ virBitmapPtr nodemask)
+{
+ virCgroupPtr cgroup_emulator = NULL;
+
+ if (!virCgroupHasController(cgroup, VIR_CGROUP_CONTROLLER_CPUSET))
+ return 0;
+
+ if (cgroup == NULL)
+ return 0; /* Not supported, so claim success */
+
+ if (virCgroupNewEmulator(cgroup, true, &cgroup_emulator) < 0)
+ goto error;
+
+ if (virCgroupMoveTask(cgroup, cgroup_emulator) < 0)
+ goto error;
+
+ if (virCgroupHasController(cgroup_emulator, VIR_CGROUP_CONTROLLER_CPUSET) &&
+ virLXCCgroupSetupCpusetTuneForEmulator(def, cgroup_emulator, nodemask) < 0)
+ goto error;
+
+ virCgroupFree(&cgroup_emulator);
+ return 0;
+
+ error:
+
+ if (cgroup_emulator) {
+ virCgroupRemove(cgroup_emulator);
+ virCgroupFree(&cgroup_emulator);
+ }
+
+ return -1;
+}
diff --git a/src/lxc/lxc_cgroup.h b/src/lxc/lxc_cgroup.h
index 0e78126..32086c5 100644
--- a/src/lxc/lxc_cgroup.h
+++ b/src/lxc/lxc_cgroup.h
@@ -33,6 +33,10 @@ int virLXCCgroupSetup(virDomainDefPtr def,
virCgroupPtr cgroup,
virBitmapPtr nodemask);
+int virLXCCgroupSetupForEmulator(virDomainDefPtr def,
+ virCgroupPtr cgroup,
+ virBitmapPtr nodemask);
+
int virLXCCgroupGetMeminfo(virLXCMeminfoPtr meminfo);
int
diff --git a/src/lxc/lxc_controller.c b/src/lxc/lxc_controller.c
index 1861dd6..1a62e20 100644
--- a/src/lxc/lxc_controller.c
+++ b/src/lxc/lxc_controller.c
@@ -698,6 +698,10 @@ static int virLXCControllerSetupResourceLimits(virLXCControllerPtr
ctrl)
if (virLXCCgroupSetup(ctrl->def, ctrl->cgroup, nodemask) < 0)
goto cleanup;
+ VIR_DEBUG("Setting cgroup for lxc emulator");
+ if (virLXCCgroupSetupForEmulator(ctrl->def, ctrl->cgroup, nodemask) < 0)
+ goto cleanup;
+
ret = 0;
cleanup:
virBitmapFree(nodemask);
--
1.7.12.4
10:34 a.m.
New subject: [libvirt] [PATCH 1/3] lxc: Implement pin emulator for container startup
On 09/04/2014 03:52 AM, Wang Rui wrote:
From: Yue Wenyuan <yuewenyuan(a)huawei.com>
This patch implements libvirt_lxc process pin with emulatorpin
specified in xml.
Signed-off-by: Wang Rui <moon.wangrui(a)huawei.com>
Signed-off-by: Yue Wenyuan <yuewenyuan(a)huawei.com>
---
src/lxc/lxc_cgroup.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++
src/lxc/lxc_cgroup.h | 4 +++
src/lxc/lxc_controller.c | 4 +++
3 files changed, 76 insertions(+)
I'm not an LXC expert, but I'll give this a go since it's been sitting
around a while.
I am curious why only the CPUSET (eg, <vcpuset ... cpuset="1-4,^3,6"> or
<emulatorpin cpuset="1-3"/>) were handled and the <emulator_period>
and
<emulator_quota> were not?
diff --git a/src/lxc/lxc_cgroup.c b/src/lxc/lxc_cgroup.c
index f9af31c..f696bf8 100644
--- a/src/lxc/lxc_cgroup.c
+++ b/src/lxc/lxc_cgroup.c
@@ -530,3 +530,71 @@ int virLXCCgroupSetup(virDomainDefPtr def,
cleanup:
return ret;
}
+
+static int virLXCCgroupSetupCpusetTuneForEmulator(virDomainDefPtr def,
Change name virLXCCgroupSetupCpusetTuneForEmulator to
virLXCSetupCgroupCpusetTunesForEmulator [1]
note the (s) on Tune as well... Could have also gone with
"CpusetCpusMems", but that really seemed too long
+ virCgroupPtr cgroup,
+ virBitmapPtr nodemask)
Although lxc_cgroup doesn't do it for the majority of the functions use:
static int
virLXCSetupCgroupEmulatorCpusetCpusMems(...)
make sure to align arguments properly too.
+{
+ int ret = -1;
+ char *mask = NULL;
+
+ if (def->placement_mode != VIR_DOMAIN_CPU_PLACEMENT_MODE_AUTO) {
+ if (def->cputune.emulatorpin) {
+ if (!(mask = virBitmapFormat(def->cputune.emulatorpin->cpumask)))
+ return ret;
+ } else if (def->cpumask) {
+ if (!(mask = virBitmapFormat(def->cpumask)))
+ return ret;
+ }
At this point mask could still be NULL, thus you need a "if (mask && "
prior to call...
+ if (virCgroupSetCpusetCpus(cgroup, mask) < 0)
+ goto cleanup;
+ }
And if MODE_AUTO, then does virLXCControllerSetupCpuAffinity called
prior to this function satisfy the doc requirement to query numad? If
so, then perhaps noting that as a comment prior to this hunk of code
would be beneficial. If not, then that needs to be handled. It doesn't
seem that way since I don't see comparable code in LXC to the
qemuPrepareCpumap code that handles the host.numaCell.
Also since you're reusing 'mask' below you'll need a VIR_FREE(mask);
prior to the next set of calls; otherwise, you'll leak it.
+
+ if (virDomainNumatuneMaybeFormatNodeset(def->numatune, nodemask,
+ &mask, -1) < 0)
+ goto cleanup;
+
+ if (mask && virCgroupSetCpusetMems(cgroup, mask) < 0)
+ goto cleanup;
+
+ ret = 0;
+
+ cleanup:
+ VIR_FREE(mask);
+ return ret;
+}
+
+int virLXCCgroupSetupForEmulator(virDomainDefPtr def,
+ virCgroupPtr cgroup,
+ virBitmapPtr nodemask)
The qemu function is qemuSetupCgroupForEmulator, thus this should be
virLXCSetupCgroupForEmulator. [2]
The function arguments need to be aligned and like above:
int
virLXCSetupCgroupForEmulator(...)
+{
+ virCgroupPtr cgroup_emulator = NULL;
+
+ if (!virCgroupHasController(cgroup, VIR_CGROUP_CONTROLLER_CPUSET))
+ return 0;
+
+ if (cgroup == NULL)
+ return 0; /* Not supported, so claim success */
This seems to be superfluous as the virCgroupHasController() will return
false if (!cgroup), although I do see that it's a cut-n-paste of the
similar qemu code.
+
+ if (virCgroupNewEmulator(cgroup, true, &cgroup_emulator) < 0)
+ goto error;
+
+ if (virCgroupMoveTask(cgroup, cgroup_emulator) < 0)
+ goto error;
+
+ if (virCgroupHasController(cgroup_emulator, VIR_CGROUP_CONTROLLER_CPUSET) &&
^^^^^^^^^^^^^^^
qemu code uses priv->cgroup, so should this code use 'cgroup' instead?
or is the qemu code incorrect and should be fixed?
+ virLXCCgroupSetupCpusetTuneForEmulator(def, cgroup_emulator,
nodemask) < 0)
[1] Use new name...
+ goto error;
+
+ virCgroupFree(&cgroup_emulator);
+ return 0;
+
+ error:
+
+ if (cgroup_emulator) {
+ virCgroupRemove(cgroup_emulator);
+ virCgroupFree(&cgroup_emulator);
+ }
+
+ return -1;
+}
diff --git a/src/lxc/lxc_cgroup.h b/src/lxc/lxc_cgroup.h
index 0e78126..32086c5 100644
--- a/src/lxc/lxc_cgroup.h
+++ b/src/lxc/lxc_cgroup.h
@@ -33,6 +33,10 @@ int virLXCCgroupSetup(virDomainDefPtr def,
virCgroupPtr cgroup,
virBitmapPtr nodemask);
+int virLXCCgroupSetupForEmulator(virDomainDefPtr def,
+ virCgroupPtr cgroup,
+ virBitmapPtr nodemask);
+
[2] See note above about name change s/CgroupSetup/SetupCgroup
Also the argument alignment is off - yes I see the previous function
prototype is too, but lets at least do this one right and fix the other
one later in either a follow up or separate patch prior to this patch.
int virLXCCgroupGetMeminfo(virLXCMeminfoPtr meminfo);
int
diff --git a/src/lxc/lxc_controller.c b/src/lxc/lxc_controller.c
index 1861dd6..1a62e20 100644
--- a/src/lxc/lxc_controller.c
+++ b/src/lxc/lxc_controller.c
@@ -698,6 +698,10 @@ static int virLXCControllerSetupResourceLimits(virLXCControllerPtr
ctrl)
if (virLXCCgroupSetup(ctrl->def, ctrl->cgroup, nodemask) < 0)
goto cleanup;
+ VIR_DEBUG("Setting cgroup for lxc emulator");
+ if (virLXCCgroupSetupForEmulator(ctrl->def, ctrl->cgroup, nodemask) < 0)
[2] See note above about name change s/CgroupSetup/SetupCgroup
+ goto cleanup;
+
ret = 0;
cleanup:
virBitmapFree(nodemask);
You need to update docs/formatdomain.html.in as well to add the LXC
Since 1.2.x (whenever this gets in) to various bits and parts of the
<cputune> and <vcpus> descriptions.
John
8:48 a.m.
New subject: [libvirt] [PATCH 1/3] lxc: Implement pin emulator for container startup
On 2014/10/22 23:34, John Ferlan wrote:
On 09/04/2014 03:52 AM, Wang Rui wrote:
> From: Yue Wenyuan <yuewenyuan(a)huawei.com>
>
> This patch implements libvirt_lxc process pin with emulatorpin
> specified in xml.
>
> Signed-off-by: Wang Rui <moon.wangrui(a)huawei.com>
> Signed-off-by: Yue Wenyuan <yuewenyuan(a)huawei.com>
> ---
> src/lxc/lxc_cgroup.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++
> src/lxc/lxc_cgroup.h | 4 +++
> src/lxc/lxc_controller.c | 4 +++
> 3 files changed, 76 insertions(+)
>
I'm not an LXC expert, but I'll give this a go since it's been sitting
around a while.
I am curious why only the CPUSET (eg, <vcpuset ... cpuset="1-4,^3,6"> or
<emulatorpin cpuset="1-3"/>) were handled and the <emulator_period>
and
<emulator_quota> were not?
Hi, John. Thank you for your review.
These patches are only the implementation of emulator pin. It is ture
that <emulator_period> and <emulator_quota> are not supported by lxc. I
think It can be handled in the later patch if these patches are ACKed.
[...]
> +{
> + int ret = -1;
> + char *mask = NULL;
> +
> + if (def->placement_mode != VIR_DOMAIN_CPU_PLACEMENT_MODE_AUTO) {
> + if (def->cputune.emulatorpin) {
> + if (!(mask = virBitmapFormat(def->cputune.emulatorpin->cpumask)))
> + return ret;
> + } else if (def->cpumask) {
> + if (!(mask = virBitmapFormat(def->cpumask)))
> + return ret;
> + }
At this point mask could still be NULL, thus you need a "if (mask && "
prior to call...
> + if (virCgroupSetCpusetCpus(cgroup, mask) < 0)
> + goto cleanup;
> + }
And if MODE_AUTO, then does virLXCControllerSetupCpuAffinity called
prior to this function satisfy the doc requirement to query numad? If
so, then perhaps noting that as a comment prior to this hunk of code
would be beneficial. If not, then that needs to be handled. It doesn't
seem that way since I don't see comparable code in LXC to the
qemuPrepareCpumap code that handles the host.numaCell.
Thanks to point that. I think virLXCControllerGetNumadAdvice which is
called in virLXCControllerSetupResourceLimits is the comparable function
to qemuPrepareCpumap. Maybe it's another bug.
Also since you're reusing 'mask' below you'll need a
VIR_FREE(mask);
prior to the next set of calls; otherwise, you'll leak it.
> +
> + if (virDomainNumatuneMaybeFormatNodeset(def->numatune, nodemask,
> + &mask, -1) < 0)
> + goto cleanup;
> +
> + if (mask && virCgroupSetCpusetMems(cgroup, mask) < 0)
> + goto cleanup;
> +
> + ret = 0;
> +
> + cleanup:
> + VIR_FREE(mask);
> + return ret;
> +}
> +
> +int virLXCCgroupSetupForEmulator(virDomainDefPtr def,
> + virCgroupPtr cgroup,
> + virBitmapPtr nodemask)
[...]
Also the argument alignment is off - yes I see the previous function
prototype is too, but lets at least do this one right and fix the other
one later in either a follow up or separate patch prior to this patch.
Some code is not well aligned. I can fix that as your suggestion.
I'll send patches V2 after release 1.2.10.
2:52 a.m.
New subject: [libvirt] [PATCH 2/3] lxc: Implement emulator pin API in lxc driver
From: Yue Wenyuan <yuewenyuan(a)huawei.com>
Implement the lxc driver method for virDomainPinEmulator
to set container's cpuset.
Signed-off-by: Wang Rui <moon.wangrui(a)huawei.com>
Signed-off-by: Yue Wenyuan <yuewenyuan(a)huawei.com>
---
src/lxc/lxc_cgroup.c | 20 ++++++++
src/lxc/lxc_cgroup.h | 3 ++
src/lxc/lxc_driver.c | 131 +++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 154 insertions(+)
diff --git a/src/lxc/lxc_cgroup.c b/src/lxc/lxc_cgroup.c
index f696bf8..7dc7c9b 100644
--- a/src/lxc/lxc_cgroup.c
+++ b/src/lxc/lxc_cgroup.c
@@ -531,6 +531,26 @@ int virLXCCgroupSetup(virDomainDefPtr def,
return ret;
}
+int
+virLXCSetupCgroupEmulatorPin(virCgroupPtr cgroup,
+ virBitmapPtr cpumask)
+{
+ int ret = -1;
+ char *new_cpus = NULL;
+
+ if (!(new_cpus = virBitmapFormat(cpumask)))
+ goto cleanup;
+
+ if (virCgroupSetCpusetCpus(cgroup, new_cpus) < 0)
+ goto cleanup;
+
+ ret = 0;
+
+ cleanup:
+ VIR_FREE(new_cpus);
+ return ret;
+}
+
static int virLXCCgroupSetupCpusetTuneForEmulator(virDomainDefPtr def,
virCgroupPtr cgroup,
virBitmapPtr nodemask)
diff --git a/src/lxc/lxc_cgroup.h b/src/lxc/lxc_cgroup.h
index 32086c5..8011a32 100644
--- a/src/lxc/lxc_cgroup.h
+++ b/src/lxc/lxc_cgroup.h
@@ -33,6 +33,9 @@ int virLXCCgroupSetup(virDomainDefPtr def,
virCgroupPtr cgroup,
virBitmapPtr nodemask);
+int virLXCSetupCgroupEmulatorPin(virCgroupPtr cgroup,
+ virBitmapPtr cpumask);
+
int virLXCCgroupSetupForEmulator(virDomainDefPtr def,
virCgroupPtr cgroup,
virBitmapPtr nodemask);
diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c
index 66d708a..9e3a877 100644
--- a/src/lxc/lxc_driver.c
+++ b/src/lxc/lxc_driver.c
@@ -1264,6 +1264,136 @@ lxcDomainCreateXML(virConnectPtr conn,
return lxcDomainCreateXMLWithFiles(conn, xml, 0, NULL, flags);
}
+static int
+lxcDomainPinEmulator(virDomainPtr dom,
+ unsigned char *cpumap,
+ int maplen,
+ unsigned int flags)
+{
+ virLXCDriverPtr driver = dom->conn->privateData;
+ virDomainObjPtr vm;
+ virCgroupPtr cgroup_emulator = NULL;
+ pid_t pid;
+ virDomainDefPtr persistentDef = NULL;
+ int ret = -1;
+ virLXCDomainObjPrivatePtr priv;
+ bool doReset = false;
+ virBitmapPtr pcpumap = NULL;
+ virLXCDriverConfigPtr cfg = NULL;
+ virCapsPtr caps = NULL;
+
+ virCheckFlags(VIR_DOMAIN_AFFECT_LIVE |
+ VIR_DOMAIN_AFFECT_CONFIG, -1);
+
+ cfg = virLXCDriverGetConfig(driver);
+
+ if (!(vm = lxcDomObjFromDomain(dom)))
+ goto cleanup;
+
+ if (virDomainPinEmulatorEnsureACL(dom->conn, vm->def, flags) < 0)
+ goto cleanup;
+
+ if (!(caps = virLXCDriverGetCapabilities(driver, false)))
+ goto cleanup;
+
+ if (vm->def->placement_mode == VIR_DOMAIN_CPU_PLACEMENT_MODE_AUTO) {
+ virReportError(VIR_ERR_OPERATION_INVALID, "%s",
+ _("Changing affinity for emulator thread dynamically "
+ "is not allowed when CPU placement is
'auto'"));
+ goto cleanup;
+ }
+
+ if (virDomainLiveConfigHelperMethod(caps, driver->xmlopt, vm, &flags,
+ &persistentDef) < 0)
+ goto cleanup;
+
+ priv = vm->privateData;
+
+ pcpumap = virBitmapNewData(cpumap, maplen);
+ if (!pcpumap)
+ goto cleanup;
+
+ if (virBitmapIsAllClear(pcpumap)) {
+ virReportError(VIR_ERR_INVALID_ARG, "%s",
+ _("Empty cpu list for pinning"));
+ goto cleanup;
+ }
+
+ if (virBitmapIsAllSet(pcpumap))
+ doReset = true;
+
+ pid = vm->pid;
+
+ if (flags & VIR_DOMAIN_AFFECT_LIVE) {
+ if (virCgroupHasController(priv->cgroup,
+ VIR_CGROUP_CONTROLLER_CPUSET)) {
+ if (virCgroupNewEmulator(priv->cgroup, false, &cgroup_emulator) <
0)
+ goto cleanup;
+ if (virLXCSetupCgroupEmulatorPin(cgroup_emulator,
+ pcpumap) < 0) {
+ virReportError(VIR_ERR_OPERATION_INVALID, "%s",
+ _("failed to set cpuset.cpus in cgroup "
+ "for lxc emulator"));
+ goto cleanup;
+ }
+ } else {
+ virReportError(VIR_ERR_SYSTEM_ERROR, "%s",
+ _("there is not cgroup controller for "
+ "lxc emulator"));
+ goto cleanup;
+ }
+ if (doReset) {
+ if (virDomainEmulatorPinDel(vm->def) < 0) {
+ virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+ _("failed to delete emulatorpin xml of "
+ "a running domain"));
+ goto cleanup;
+ }
+ } else {
+ if (virDomainEmulatorPinAdd(vm->def, cpumap, maplen) < 0) {
+ virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+ _("failed to update or add emulatorpin xml "
+ "of a running domain"));
+ goto cleanup;
+ }
+ }
+ if (virDomainSaveStatus(driver->xmlopt, cfg->stateDir, vm) < 0)
+ goto cleanup;
+ }
+
+ if (flags & VIR_DOMAIN_AFFECT_CONFIG) {
+ if (doReset) {
+ if (virDomainEmulatorPinDel(persistentDef) < 0) {
+ virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+ _("failed to delete emulatorpin xml of "
+ "a persistent domain"));
+ goto cleanup;
+ }
+ } else {
+ if (virDomainEmulatorPinAdd(persistentDef, cpumap, maplen) < 0) {
+ virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+ _("failed to update or add emulatorpin xml "
+ "of a persistent domain"));
+ goto cleanup;
+ }
+ }
+
+ ret = virDomainSaveConfig(cfg->configDir, persistentDef);
+ goto cleanup;
+ }
+
+ ret = 0;
+
+ cleanup:
+ if (cgroup_emulator)
+ virCgroupFree(&cgroup_emulator);
+ virBitmapFree(pcpumap);
+ virObjectUnref(caps);
+ if (vm)
+ virObjectUnlock(vm);
+ virObjectUnref(cfg);
+ return ret;
+}
static int lxcDomainGetSecurityLabel(virDomainPtr dom, virSecurityLabelPtr seclabel)
{
@@ -5700,6 +5830,7 @@ static virDriver lxcDriver = {
.connectListAllDomains = lxcConnectListAllDomains, /* 0.9.13 */
.domainCreateXML = lxcDomainCreateXML, /* 0.4.4 */
.domainCreateXMLWithFiles = lxcDomainCreateXMLWithFiles, /* 1.1.1 */
+ .domainPinEmulator = lxcDomainPinEmulator, /*1.2.9*/
.domainLookupByID = lxcDomainLookupByID, /* 0.4.2 */
.domainLookupByUUID = lxcDomainLookupByUUID, /* 0.4.2 */
.domainLookupByName = lxcDomainLookupByName, /* 0.4.2 */
--
1.7.12.4
10:34 a.m.
New subject: [libvirt] [PATCH 2/3] lxc: Implement emulator pin API in lxc driver
On 09/04/2014 03:52 AM, Wang Rui wrote:
From: Yue Wenyuan <yuewenyuan(a)huawei.com>
Implement the lxc driver method for virDomainPinEmulator
to set container's cpuset.
Signed-off-by: Wang Rui <moon.wangrui(a)huawei.com>
Signed-off-by: Yue Wenyuan <yuewenyuan(a)huawei.com>
---
src/lxc/lxc_cgroup.c | 20 ++++++++
src/lxc/lxc_cgroup.h | 3 ++
src/lxc/lxc_driver.c | 131 +++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 154 insertions(+)
diff --git a/src/lxc/lxc_cgroup.c b/src/lxc/lxc_cgroup.c
index f696bf8..7dc7c9b 100644
--- a/src/lxc/lxc_cgroup.c
+++ b/src/lxc/lxc_cgroup.c
@@ -531,6 +531,26 @@ int virLXCCgroupSetup(virDomainDefPtr def,
return ret;
}
+int
+virLXCSetupCgroupEmulatorPin(virCgroupPtr cgroup,
+ virBitmapPtr cpumask)
+{
+ int ret = -1;
+ char *new_cpus = NULL;
+
+ if (!(new_cpus = virBitmapFormat(cpumask)))
+ goto cleanup;
+
+ if (virCgroupSetCpusetCpus(cgroup, new_cpus) < 0)
+ goto cleanup;
+
+ ret = 0;
+
+ cleanup:
+ VIR_FREE(new_cpus);
+ return ret;
+}
+
static int virLXCCgroupSetupCpusetTuneForEmulator(virDomainDefPtr def,
virCgroupPtr cgroup,
virBitmapPtr nodemask)
diff --git a/src/lxc/lxc_cgroup.h b/src/lxc/lxc_cgroup.h
index 32086c5..8011a32 100644
--- a/src/lxc/lxc_cgroup.h
+++ b/src/lxc/lxc_cgroup.h
@@ -33,6 +33,9 @@ int virLXCCgroupSetup(virDomainDefPtr def,
virCgroupPtr cgroup,
virBitmapPtr nodemask);
+int virLXCSetupCgroupEmulatorPin(virCgroupPtr cgroup,
+ virBitmapPtr cpumask);
+
alignment issues with the arguments
int virLXCCgroupSetupForEmulator(virDomainDefPtr def,
virCgroupPtr cgroup,
virBitmapPtr nodemask);
diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c
index 66d708a..9e3a877 100644
--- a/src/lxc/lxc_driver.c
+++ b/src/lxc/lxc_driver.c
@@ -1264,6 +1264,136 @@ lxcDomainCreateXML(virConnectPtr conn,
return lxcDomainCreateXMLWithFiles(conn, xml, 0, NULL, flags);
}
+static int
+lxcDomainPinEmulator(virDomainPtr dom,
+ unsigned char *cpumap,
+ int maplen,
+ unsigned int flags)
Alignment issues with arguments.
+{
+ virLXCDriverPtr driver = dom->conn->privateData;
+ virDomainObjPtr vm;
+ virCgroupPtr cgroup_emulator = NULL;
+ pid_t pid;
+ virDomainDefPtr persistentDef = NULL;
+ int ret = -1;
+ virLXCDomainObjPrivatePtr priv;
+ bool doReset = false;
+ virBitmapPtr pcpumap = NULL;
+ virLXCDriverConfigPtr cfg = NULL;
+ virCapsPtr caps = NULL;
+
+ virCheckFlags(VIR_DOMAIN_AFFECT_LIVE |
+ VIR_DOMAIN_AFFECT_CONFIG, -1);
+
+ cfg = virLXCDriverGetConfig(driver);
+
+ if (!(vm = lxcDomObjFromDomain(dom)))
+ goto cleanup;
+
+ if (virDomainPinEmulatorEnsureACL(dom->conn, vm->def, flags) < 0)
+ goto cleanup;
+
+ if (!(caps = virLXCDriverGetCapabilities(driver, false)))
+ goto cleanup;
+
+ if (vm->def->placement_mode == VIR_DOMAIN_CPU_PLACEMENT_MODE_AUTO) {
+ virReportError(VIR_ERR_OPERATION_INVALID, "%s",
+ _("Changing affinity for emulator thread dynamically "
+ "is not allowed when CPU placement is
'auto'"));
+ goto cleanup;
+ }
+
+ if (virDomainLiveConfigHelperMethod(caps, driver->xmlopt, vm, &flags,
+ &persistentDef) < 0)
+ goto cleanup;
+
+ priv = vm->privateData;
+
+ pcpumap = virBitmapNewData(cpumap, maplen);
+ if (!pcpumap)
+ goto cleanup;
+
+ if (virBitmapIsAllClear(pcpumap)) {
+ virReportError(VIR_ERR_INVALID_ARG, "%s",
+ _("Empty cpu list for pinning"));
+ goto cleanup;
+ }
+
+ if (virBitmapIsAllSet(pcpumap))
+ doReset = true;
+
+ pid = vm->pid;
This is set, but never used - compilation failure on my Fedora system
+
+ if (flags & VIR_DOMAIN_AFFECT_LIVE) {
+ if (virCgroupHasController(priv->cgroup,
+ VIR_CGROUP_CONTROLLER_CPUSET)) {
Alignment issues
+ if (virCgroupNewEmulator(priv->cgroup, false,
&cgroup_emulator) < 0)
+ goto cleanup;
+ if (virLXCSetupCgroupEmulatorPin(cgroup_emulator,
+ pcpumap) < 0) {
Alignment issues
+ virReportError(VIR_ERR_OPERATION_INVALID,
"%s",
+ _("failed to set cpuset.cpus in cgroup "
+ "for lxc emulator"));
+ goto cleanup;
+ }
+ } else {
And here is the missing line from the comparable qemu code, e.g. a call
to virProcessSetAffinity(pid, pcpumap).
I note that the LXC code during virLXCControllerSetupCpuAffinity will
call SetAffinity with 0 (zero) as the pid. Whether that's relevant here
or not I'm not quite sure.
+ virReportError(VIR_ERR_SYSTEM_ERROR, "%s",
+ _("there is not cgroup controller for "
+ "lxc emulator"));
Perhaps better said "no cpuset cgroup controller found for lxc emulator"
- although just because HasController/CPUSET fails, we won't continue
which doesn't seem right.
+ goto cleanup;
+ }
+ if (doReset) {
+ if (virDomainEmulatorPinDel(vm->def) < 0) {
+ virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+ _("failed to delete emulatorpin xml of "
+ "a running domain"));
+ goto cleanup;
+ }
+ } else {
+ if (virDomainEmulatorPinAdd(vm->def, cpumap, maplen) < 0) {
+ virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+ _("failed to update or add emulatorpin xml "
+ "of a running domain"));
+ goto cleanup;
+ }
+ }
+ if (virDomainSaveStatus(driver->xmlopt, cfg->stateDir, vm) < 0)
+ goto cleanup;
The qemu code will issue a (new) event here for
VIR_DOMAIN_TUNABLE_CPU_EMULATORPIN... Is that applicable here? If so and
you add it, don't forget to clean up the event during cleanup:
+ }
+
+ if (flags & VIR_DOMAIN_AFFECT_CONFIG) {
+ if (doReset) {
+ if (virDomainEmulatorPinDel(persistentDef) < 0) {
+ virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+ _("failed to delete emulatorpin xml of "
+ "a persistent domain"));
+ goto cleanup;
+ }
+ } else {
+ if (virDomainEmulatorPinAdd(persistentDef, cpumap, maplen) < 0) {
+ virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
+ _("failed to update or add emulatorpin xml "
+ "of a persistent domain"));
+ goto cleanup;
+ }
+ }
+
+ ret = virDomainSaveConfig(cfg->configDir, persistentDef);
+ goto cleanup;
+ }
+
+ ret = 0;
+
+ cleanup:
+ if (cgroup_emulator)
+ virCgroupFree(&cgroup_emulator);
+ virBitmapFree(pcpumap);
+ virObjectUnref(caps);
+ if (vm)
+ virObjectUnlock(vm);
+ virObjectUnref(cfg);
+ return ret;
+}
static int lxcDomainGetSecurityLabel(virDomainPtr dom, virSecurityLabelPtr seclabel)
{
@@ -5700,6 +5830,7 @@ static virDriver lxcDriver = {
.connectListAllDomains = lxcConnectListAllDomains, /* 0.9.13 */
.domainCreateXML = lxcDomainCreateXML, /* 0.4.4 */
.domainCreateXMLWithFiles = lxcDomainCreateXMLWithFiles, /* 1.1.1 */
+ .domainPinEmulator = lxcDomainPinEmulator, /*1.2.9*/
Comment spacing is off, will now be at least /* 1.2.10 */
John
.domainLookupByID = lxcDomainLookupByID, /* 0.4.2 */
.domainLookupByUUID = lxcDomainLookupByUUID, /* 0.4.2 */
.domainLookupByName = lxcDomainLookupByName, /* 0.4.2 */
2:52 a.m.
New subject: [libvirt] [PATCH 3/3] lxc: Implement geting emulator pin info API in lxc driver
From: Yue Wenyuan <yuewenyuan(a)huawei.com>
Implement the lxc driver method for virDomainGetEmulatorPinInfo
to get container's cpuset.
Signed-off-by: Wang Rui <moon.wangrui(a)huawei.com>
Signed-off-by: Yue Wenyuan <yuewenyuan(a)huawei.com>
---
src/lxc/lxc_driver.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 75 insertions(+)
diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c
index 9e3a877..6694cf3 100644
--- a/src/lxc/lxc_driver.c
+++ b/src/lxc/lxc_driver.c
@@ -1395,6 +1395,80 @@ lxcDomainPinEmulator(virDomainPtr dom,
return ret;
}
+static int
+lxcDomainGetEmulatorPinInfo(virDomainPtr dom,
+ unsigned char *cpumaps,
+ int maplen,
+ unsigned int flags)
+{
+ virLXCDriverPtr driver = dom->conn->privateData;
+ virDomainObjPtr vm = NULL;
+ virDomainDefPtr targetDef = NULL;
+ int ret = -1;
+ int maxcpu, hostcpus, pcpu;
+ virBitmapPtr cpumask = NULL;
+ bool pinned;
+ virCapsPtr caps = NULL;
+
+ virCheckFlags(VIR_DOMAIN_AFFECT_LIVE |
+ VIR_DOMAIN_AFFECT_CONFIG, -1);
+
+ if (!(vm = lxcDomObjFromDomain(dom)))
+ goto cleanup;
+
+ if (virDomainGetEmulatorPinInfoEnsureACL(dom->conn, vm->def) < 0)
+ goto cleanup;
+
+ if (!(caps = virLXCDriverGetCapabilities(driver, false)))
+ goto cleanup;
+
+ if (virDomainLiveConfigHelperMethod(caps, driver->xmlopt,
+ vm, &flags, &targetDef) < 0)
+ goto cleanup;
+
+ if (flags & VIR_DOMAIN_AFFECT_LIVE)
+ targetDef = vm->def;
+
+ sa_assert(targetDef);
+
+ if ((hostcpus = nodeGetCPUCount()) < 0)
+ goto cleanup;
+
+ maxcpu = maplen * 8;
+ if (maxcpu > hostcpus)
+ maxcpu = hostcpus;
+
+ /* initialize cpumaps */
+ memset(cpumaps, 0xff, maplen);
+ if (maxcpu % 8) {
+ cpumaps[maplen - 1] &= (1 << maxcpu % 8) - 1;
+ }
+
+ if (targetDef->cputune.emulatorpin) {
+ cpumask = targetDef->cputune.emulatorpin->cpumask;
+ } else if (targetDef->cpumask) {
+ cpumask = targetDef->cpumask;
+ } else {
+ ret = 0;
+ goto cleanup;
+ }
+
+ for (pcpu = 0; pcpu < maxcpu; pcpu++) {
+ if (virBitmapGetBit(cpumask, pcpu, &pinned) < 0)
+ goto cleanup;
+ if (!pinned)
+ VIR_UNUSE_CPU(cpumaps, pcpu);
+ }
+
+ ret = 1;
+
+ cleanup:
+ if (vm)
+ virObjectUnlock(vm);
+ virObjectUnref(caps);
+ return ret;
+}
+
static int lxcDomainGetSecurityLabel(virDomainPtr dom, virSecurityLabelPtr seclabel)
{
virLXCDriverPtr driver = dom->conn->privateData;
@@ -5831,6 +5905,7 @@ static virDriver lxcDriver = {
.domainCreateXML = lxcDomainCreateXML, /* 0.4.4 */
.domainCreateXMLWithFiles = lxcDomainCreateXMLWithFiles, /* 1.1.1 */
.domainPinEmulator = lxcDomainPinEmulator, /*1.2.9*/
+ .domainGetEmulatorPinInfo = lxcDomainGetEmulatorPinInfo, /*1.2.9*/
.domainLookupByID = lxcDomainLookupByID, /* 0.4.2 */
.domainLookupByUUID = lxcDomainLookupByUUID, /* 0.4.2 */
.domainLookupByName = lxcDomainLookupByName, /* 0.4.2 */
--
1.7.12.4
10:34 a.m.
New subject: [libvirt] [PATCH 3/3] lxc: Implement geting emulator pin info API in lxc driver
Fix the $SUBJ "s/geting/getting"
On 09/04/2014 03:52 AM, Wang Rui wrote:
From: Yue Wenyuan <yuewenyuan(a)huawei.com>
Implement the lxc driver method for virDomainGetEmulatorPinInfo
to get container's cpuset.
Signed-off-by: Wang Rui <moon.wangrui(a)huawei.com>
Signed-off-by: Yue Wenyuan <yuewenyuan(a)huawei.com>
---
src/lxc/lxc_driver.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 75 insertions(+)
diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c
index 9e3a877..6694cf3 100644
--- a/src/lxc/lxc_driver.c
+++ b/src/lxc/lxc_driver.c
@@ -1395,6 +1395,80 @@ lxcDomainPinEmulator(virDomainPtr dom,
return ret;
}
Since the following is essentially a copy of the qemu function with the
only changes being "LXC" for "QEMU" or "lxc" for
"qemu" - it seems that
it should be possible to create some sort of common vir/util function
especially for all the lines between [1] and [2] below. Of course
that's probably true in other areas, but since I was comparing here I
guess it
+static int
+lxcDomainGetEmulatorPinInfo(virDomainPtr dom,
+ unsigned char *cpumaps,
+ int maplen,
+ unsigned int flags)
+{
+ virLXCDriverPtr driver = dom->conn->privateData;
+ virDomainObjPtr vm = NULL;
+ virDomainDefPtr targetDef = NULL;
+ int ret = -1;
+ int maxcpu, hostcpus, pcpu;
+ virBitmapPtr cpumask = NULL;
+ bool pinned;
+ virCapsPtr caps = NULL;
+
+ virCheckFlags(VIR_DOMAIN_AFFECT_LIVE |
+ VIR_DOMAIN_AFFECT_CONFIG, -1);
+
+ if (!(vm = lxcDomObjFromDomain(dom)))
+ goto cleanup;
+
+ if (virDomainGetEmulatorPinInfoEnsureACL(dom->conn, vm->def) < 0)
+ goto cleanup;
+
+ if (!(caps = virLXCDriverGetCapabilities(driver, false)))
+ goto cleanup;
+
+ if (virDomainLiveConfigHelperMethod(caps, driver->xmlopt,
+ vm, &flags, &targetDef) < 0)
+ goto cleanup;
+
[1]
+ if (flags & VIR_DOMAIN_AFFECT_LIVE)
+ targetDef = vm->def;
+
+ sa_assert(targetDef);
+
+ if ((hostcpus = nodeGetCPUCount()) < 0)
+ goto cleanup;
+
+ maxcpu = maplen * 8;
+ if (maxcpu > hostcpus)
+ maxcpu = hostcpus;
+
+ /* initialize cpumaps */
+ memset(cpumaps, 0xff, maplen);
+ if (maxcpu % 8) {
+ cpumaps[maplen - 1] &= (1 << maxcpu % 8) - 1;
+ }
+
+ if (targetDef->cputune.emulatorpin) {
+ cpumask = targetDef->cputune.emulatorpin->cpumask;
+ } else if (targetDef->cpumask) {
+ cpumask = targetDef->cpumask;
+ } else {
+ ret = 0;
+ goto cleanup;
+ }
+
+ for (pcpu = 0; pcpu < maxcpu; pcpu++) {
+ if (virBitmapGetBit(cpumask, pcpu, &pinned) < 0)
+ goto cleanup;
+ if (!pinned)
+ VIR_UNUSE_CPU(cpumaps, pcpu);
+ }
[2]
+
+ ret = 1;
+
+ cleanup:
+ if (vm)
+ virObjectUnlock(vm);
+ virObjectUnref(caps);
+ return ret;
+}
+
static int lxcDomainGetSecurityLabel(virDomainPtr dom, virSecurityLabelPtr seclabel)
{
virLXCDriverPtr driver = dom->conn->privateData;
@@ -5831,6 +5905,7 @@ static virDriver lxcDriver = {
.domainCreateXML = lxcDomainCreateXML, /* 0.4.4 */
.domainCreateXMLWithFiles = lxcDomainCreateXMLWithFiles, /* 1.1.1 */
.domainPinEmulator = lxcDomainPinEmulator, /*1.2.9*/
+ .domainGetEmulatorPinInfo = lxcDomainGetEmulatorPinInfo, /*1.2.9*/
Comment spacing is off, will now be at least /* 1.2.10 */
John
.domainLookupByID = lxcDomainLookupByID, /* 0.4.2 */
.domainLookupByUUID = lxcDomainLookupByUUID, /* 0.4.2 */
.domainLookupByName = lxcDomainLookupByName, /* 0.4.2 */
12:58 a.m.
Ping?
On 2014/9/4 15:52, Wang Rui wrote:
We can specify cpuset for a container defined with the xml
like <vcpu placement='static' cpuset='0-3'> to achieve cpu
isolation. It works when container is started. But there
is no implements we can use to either change or get cpuset.
The following patches implement the lxc driver methods for
virDomainPinEmulator and virDomainGetEmulatorPinInfo. Also
support container startup with emulator affinity info in xml.
After these patches, we can set and get libvirt_lxc cpuset.
Yue Wenyuan (3):
lxc: Implement pin emulator for container startup
lxc: Implement emulator pin API in lxc driver
lxc: Implement geting emulator pin info API in lxc driver
src/lxc/lxc_cgroup.c | 88 ++++++++++++++++++++
src/lxc/lxc_cgroup.h | 7 ++
src/lxc/lxc_controller.c | 4 +
src/lxc/lxc_driver.c | 206 +++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 305 insertions(+)
1:14 a.m.
On Fri, 2014-10-10 at 13:58 +0800, Wang Rui wrote:
Ping?
I found some wrong indentation in the 3 patches.
like following in patch 1/3:
+static int virLXCCgroupSetupCpusetTuneForEmulator(virDomainDefPtr def,
+ virCgroupPtr cgroup,
+ virBitmapPtr nodemask)
Thanks,
Chen
On 2014/9/4 15:52, Wang Rui wrote:
> We can specify cpuset for a container defined with the xml
> like <vcpu placement='static' cpuset='0-3'> to achieve cpu
> isolation. It works when container is started. But there
> is no implements we can use to either change or get cpuset.
>
> The following patches implement the lxc driver methods for
> virDomainPinEmulator and virDomainGetEmulatorPinInfo. Also
> support container startup with emulator affinity info in xml.
>
> After these patches, we can set and get libvirt_lxc cpuset.
>
> Yue Wenyuan (3):
> lxc: Implement pin emulator for container startup
> lxc: Implement emulator pin API in lxc driver
> lxc: Implement geting emulator pin info API in lxc driver
>
> src/lxc/lxc_cgroup.c | 88 ++++++++++++++++++++
> src/lxc/lxc_cgroup.h | 7 ++
> src/lxc/lxc_controller.c | 4 +
> src/lxc/lxc_driver.c | 206 +++++++++++++++++++++++++++++++++++++++++++++++
> 4 files changed, 305 insertions(+)
>
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
3710
days inactive
3764
days old
9 comments
3 participants
participants (3)
-
Chen, Fan -
John Ferlan -
Wang Rui