If the memory mode is specified as preferred, we get the following error when starting domain. error: Unable to write to '$my_cgroup_path/cpuset.mems': Device or resource busy XML is configured with numatune as follows: <numatune> <memory mode='preferred' nodeset='0'/> </numatune> If memory mode is 'preferred', cpuset.mems in cgroup shouldn't be set to 'nodeset'. I find that maybe commit 1a7be8c600905aa07ac2d78293336ba8523ad48e changes the former logic of checking mode in virDomainNumatuneGetNodeset. Signed-off-by: Wang Rui <moon.wangrui(a)huawei.com&gt; --- src/qemu/qemu_cgroup.c | 5 +++++ 1 file changed, 5 insertions(+)

diff --git a/src/qemu/qemu_cgroup.c b/src/qemu/qemu_cgroup.c index b5bdb36..8685d6f 100644 --- a/src/qemu/qemu_cgroup.c +++ b/src/qemu/qemu_cgroup.c @@ -618,6 +618,11 @@ qemuSetupCpusetMems(virDomainObjPtr vm, if (!virCgroupHasController(priv->cgroup, VIR_CGROUP_CONTROLLER_CPUSET)) return 0; + if (virDomainNumatuneGetMode(vm->def->numatune, -1) != + VIR_DOMAIN_NUMATUNE_MEM_STRICT) { + return 0; + } +

On 2014/11/4 22:04, Martin Kletzander wrote: > On Tue, Nov 04, 2014 at 09:22:22PM +0800, Wang Rui wrote: >> If the memory mode is specified as preferred, we get the following error when >> starting domain. >> >> error: Unable to write to '$my_cgroup_path/cpuset.mems': Device or resource busy >> >> XML is configured with numatune as follows: >> <numatune> >> <memory mode='preferred' nodeset='0'/> >> </numatune> >> >> If memory mode is 'preferred', cpuset.mems in cgroup shouldn't be set to >> 'nodeset'. I find that maybe commit 1a7be8c600905aa07ac2d78293336ba8523ad48e >> changes the former logic of checking mode in virDomainNumatuneGetNodeset. >> >> Signed-off-by: Wang Rui <moon.wangrui(a)huawei.com&gt; >> --- >> src/qemu/qemu_cgroup.c | 5 +++++ >> 1 file changed, 5 insertions(+) >> > > Thanks for catching that, it definitely is a problem, but I think it > is cause by commit 93e82727ec11d471d2ef3a18835e1fdfe062cef1. > > It should be also fixed in virLXCCgroupSetupCpusetTune() for LXC. > OK. I'll try to fix it for LXC in another patch.

>> diff --git a/src/qemu/qemu_cgroup.c b/src/qemu/qemu_cgroup.c >> index b5bdb36..8685d6f 100644 >> --- a/src/qemu/qemu_cgroup.c >> +++ b/src/qemu/qemu_cgroup.c >> @@ -618,6 +618,11 @@ qemuSetupCpusetMems(virDomainObjPtr vm, >> if (!virCgroupHasController(priv->cgroup, VIR_CGROUP_CONTROLLER_CPUSET)) >> return 0; >> >> + if (virDomainNumatuneGetMode(vm->def->numatune, -1) != >> + VIR_DOMAIN_NUMATUNE_MEM_STRICT) { >> + return 0; >> + } >> + > > One question, is it problem only for 'preferred' or 'interleaved' as > well? Because if it's only problem for 'preferred', then the check is > wrong. If it's problem for 'interleaved' as well, then the commit > message is wrong. > 'interleave' with a single node(such as nodeset='0') will cause the same error. But 'interleave' mode should not live with a single node. So maybe there's another bugfix to check 'interleave' with single node.

If configured with 'interleave' and multiple nodes(such as nodeset='0-1'), VM can be started successfully. And cpuset.mems is set to the same nodeset. So I'll revise my patch. I'll send patches V2. Conclusion: 1/3 : add check for 'interleave' mode with single numa node 2/3 : fix this problem in qemu 3/3 : fix this problem in lxc Is it OK? > Anyway, after either one is fixed, I can push this. > > Thank you, > Martin

On 2014/11/5 16:07, Martin Kletzander wrote: [...] >>>> diff --git a/src/qemu/qemu_cgroup.c b/src/qemu/qemu_cgroup.c >>>> index b5bdb36..8685d6f 100644 >>>> --- a/src/qemu/qemu_cgroup.c >>>> +++ b/src/qemu/qemu_cgroup.c >>>> @@ -618,6 +618,11 @@ qemuSetupCpusetMems(virDomainObjPtr vm, >>>> if (!virCgroupHasController(priv->cgroup, VIR_CGROUP_CONTROLLER_CPUSET)) >>>> return 0; >>>> >>>> + if (virDomainNumatuneGetMode(vm->def->numatune, -1) != >>>> + VIR_DOMAIN_NUMATUNE_MEM_STRICT) { >>>> + return 0; >>>> + } >>>> + >>> >>> One question, is it problem only for 'preferred' or 'interleaved' as >>> well? Because if it's only problem for 'preferred', then the check is >>> wrong. If it's problem for 'interleaved' as well, then the commit >>> message is wrong. >>> >> 'interleave' with a single node(such as nodeset='0') will cause the same error. >> But 'interleave' mode should not live with a single node. So maybe there's >> another bugfix to check 'interleave' with single node. >> > > Well, I'd be OK with just changing the commit message to mention that. > This fix is still a valid one and will fix both issues, won't it? > >> If configured with 'interleave' and multiple nodes(such as nodeset='0-1'), >> VM can be started successfully. And cpuset.mems is set to the same nodeset. >> So I'll revise my patch. >> >> I'll send patches V2. Conclusion: >> >> 1/3 : add check for 'interleave' mode with single numa node >> 2/3 : fix this problem in qemu >> 3/3 : fix this problem in lxc >> >> Is it OK? >> >>> Anyway, after either one is fixed, I can push this. >>> I tested this problem again and found that this error occurred with each memory mode. It is broke by commit 411cea638f6ec8503b7142a31e58b1cd85dbeaba which is produced by me. qemu: move setting emulatorpin ahead of monitor showing up I'm sorry for that. That patch moved qemuSetupCgroupForEmulator before qemuSetupCgroupPostInit. I have ideas to fix that. 1. Move qemuSetupCgroupPostInit ahead of monitor showing up, too. Of course it's before qemuSetupCgroupForEmulator. This action to fix the bug which is introduced by me. (RFC)

2. Anyway the first problem is fixed, I have found the second problem which is I wanted to fix originally. If memory mode is 'preferred' and with one node (such as nodeset='0'), domain's memory is not in node 0 absolutely. Assumption that node 0 doesn't have enough memory, memory can be allocated on node 1. Then if we set cpuset.mems to '0', it may cause OOM. The solution is checking memory mode in (lxc)qemuSetupCpusetMems as my patch on Tuesday. Such as + if (virDomainNumatuneGetMode(vm->def->numatune, -1) != + VIR_DOMAIN_NUMATUNE_MEM_PREFERRED) {

BTW: 3. After the first problem has been fixed, we can start domains with xml: <numatune> <memory mode='interleave' nodeset='0'/> </numatune> Is a single node '0' valid for 'interleave' ? I take 'interleave' as 'at least two nodes'.