[libvirt] [PATCH] Host cpu offline/online removes the cpus available for vms
Online/Offline operations on the host cpus removes the machine/cpuset.cpus which are never added back. The guests with vcpu pinning can fail to boot unless the xml is edited. If the possibility that the offlined cpus are onlined back, the cpuset.cpus can be updated upon start of the guest thus allowing the guests to boot back without erroring out. Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> --- src/util/vircgroup.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/src/util/vircgroup.c b/src/util/vircgroup.c index a6d60c5..52575c9 100644 --- a/src/util/vircgroup.c +++ b/src/util/vircgroup.c @@ -72,6 +72,7 @@ typedef enum { * before creating subcgroups and * attaching tasks */ + VIR_CGROUP_CPUS_HIERACHY = 1 << 1, /* call virCgroupCpuSetInherit */ } virCgroupFlags; @@ -891,6 +892,7 @@ virCgroupMakeGroup(virCgroupPtr parent, VIR_DEBUG("Make group %s", group->path); for (i = 0; i < VIR_CGROUP_CONTROLLER_LAST; i++) { char *path = NULL; + int inheritCpuset = 0; /* We must never mkdir() in systemd's hierarchy */ if (i == VIR_CGROUP_CONTROLLER_SYSTEMD) { @@ -933,15 +935,7 @@ virCgroupMakeGroup(virCgroupPtr parent, goto cleanup; } } - if (group->controllers[VIR_CGROUP_CONTROLLER_CPUSET].mountPoint != NULL && - (i == VIR_CGROUP_CONTROLLER_CPUSET || - STREQ(group->controllers[i].mountPoint, - group->controllers[VIR_CGROUP_CONTROLLER_CPUSET].mountPoint))) { - if (virCgroupCpuSetInherit(parent, group) < 0) { - VIR_FREE(path); - goto cleanup; - } - } + inheritCpuset = 1; /* * Note that virCgroupSetMemoryUseHierarchy should always be * called prior to creating subcgroups and attaching tasks. @@ -958,6 +952,19 @@ virCgroupMakeGroup(virCgroupPtr parent, } } + if (inheritCpuset || (flags & VIR_CGROUP_CPUS_HIERACHY)) + { + if (group->controllers[VIR_CGROUP_CONTROLLER_CPUSET].mountPoint != NULL && + (i == VIR_CGROUP_CONTROLLER_CPUSET || + STREQ(group->controllers[i].mountPoint, + group->controllers[VIR_CGROUP_CONTROLLER_CPUSET].mountPoint))) { + if (virCgroupCpuSetInherit(parent, group) < 0) { + VIR_FREE(path); + goto cleanup; + } + } + } + VIR_FREE(path); } @@ -1275,7 +1282,7 @@ virCgroupNewPartition(const char *path, if (virCgroupNew(-1, parentPath, NULL, controllers, &parent) < 0) goto cleanup; - if (virCgroupMakeGroup(parent, *group, create, VIR_CGROUP_NONE) < 0) { + if (virCgroupMakeGroup(parent, *group, create, VIR_CGROUP_CPUS_HIERACHY) < 0) { virCgroupRemove(*group); goto cleanup; }
On 01/31/2014 12:22 AM, Shivaprasad G Bhat wrote:
Online/Offline operations on the host cpus removes the machine/cpuset.cpus which are never added back. The guests with vcpu pinning can fail to boot unless the xml is edited.
If the possibility that the offlined cpus are onlined back, the cpuset.cpus can be updated upon start of the guest thus allowing the guests to boot back without erroring out.
Please wrap commit message lines; typically, wrapping less than 70 characters since 'git log' indents the lines.
Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> --- src/util/vircgroup.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/src/util/vircgroup.c b/src/util/vircgroup.c index a6d60c5..52575c9 100644 --- a/src/util/vircgroup.c +++ b/src/util/vircgroup.c @@ -72,6 +72,7 @@ typedef enum { * before creating subcgroups and * attaching tasks */ + VIR_CGROUP_CPUS_HIERACHY = 1 << 1, /* call virCgroupCpuSetInherit */
s/HIERACHY/HIERARCHY/ I'm not sure about the technical aspect of this patch, on whether this is the best approach to deal with the situation. I think the kernel has caused several issues with how it handles offline cpu vs. cgroups, and wonder if the fix belongs in the kernel. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Thanks for the comments Eric. The offline, removes the entry from the cgroups and all its subcgroups. On Online, I guess kernel does not know till which sub-cgroup it should add the entry. I think that is how cgroup is designed to behave when it lacks the policy knowledge. Let me know what you think. Regards, Shivaprasad G Bhat On Fri, Jan 31, 2014 at 9:58 PM, Eric Blake <eblake@redhat.com> wrote:
On 01/31/2014 12:22 AM, Shivaprasad G Bhat wrote:
Online/Offline operations on the host cpus removes the machine/cpuset.cpus which are never added back. The guests with vcpu pinning can fail to boot unless the xml is edited.
If the possibility that the offlined cpus are onlined back, the cpuset.cpus can be updated upon start of the guest thus allowing the guests to boot back without erroring out.
Please wrap commit message lines; typically, wrapping less than 70 characters since 'git log' indents the lines.
Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> --- src/util/vircgroup.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/src/util/vircgroup.c b/src/util/vircgroup.c index a6d60c5..52575c9 100644 --- a/src/util/vircgroup.c +++ b/src/util/vircgroup.c @@ -72,6 +72,7 @@ typedef enum { * before creating subcgroups and * attaching tasks */ + VIR_CGROUP_CPUS_HIERACHY = 1 << 1, /* call virCgroupCpuSetInherit */
s/HIERACHY/HIERARCHY/
I'm not sure about the technical aspect of this patch, on whether this is the best approach to deal with the situation. I think the kernel has caused several issues with how it handles offline cpu vs. cgroups, and wonder if the fix belongs in the kernel.
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Hi Eric, Came across a thread where the kernel stands not to change the cpuset's behavior during the online->offline->online scenario. https://lkml.org/lkml/2012/5/4/265 Hope fixing this in Libvirt makes more sense now. Regards, Shivaprasad G Bhat On Sat, Feb 1, 2014 at 8:49 AM, Shivaprasad bhat <shivaprasadbhat@gmail.com> wrote:
Thanks for the comments Eric.
The offline, removes the entry from the cgroups and all its subcgroups. On Online, I guess kernel does not know till which sub-cgroup it should add the entry. I think that is how cgroup is designed to behave when it lacks the policy knowledge. Let me know what you think.
Regards, Shivaprasad G Bhat
On Fri, Jan 31, 2014 at 9:58 PM, Eric Blake <eblake@redhat.com> wrote:
On 01/31/2014 12:22 AM, Shivaprasad G Bhat wrote:
Online/Offline operations on the host cpus removes the machine/cpuset.cpus which are never added back. The guests with vcpu pinning can fail to boot unless the xml is edited.
If the possibility that the offlined cpus are onlined back, the cpuset.cpus can be updated upon start of the guest thus allowing the guests to boot back without erroring out.
Please wrap commit message lines; typically, wrapping less than 70 characters since 'git log' indents the lines.
Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> --- src/util/vircgroup.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/src/util/vircgroup.c b/src/util/vircgroup.c index a6d60c5..52575c9 100644 --- a/src/util/vircgroup.c +++ b/src/util/vircgroup.c @@ -72,6 +72,7 @@ typedef enum { * before creating subcgroups and * attaching tasks */ + VIR_CGROUP_CPUS_HIERACHY = 1 << 1, /* call virCgroupCpuSetInherit */
s/HIERACHY/HIERARCHY/
I'm not sure about the technical aspect of this patch, on whether this is the best approach to deal with the situation. I think the kernel has caused several issues with how it handles offline cpu vs. cgroups, and wonder if the fix belongs in the kernel.
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
participants (3)
-
Eric Blake -
Shivaprasad bhat -
Shivaprasad G Bhat