4:37 a.m.
Hello All,
Since 4.5.0-23.el7 version (Red Hat 7.7), when I launch pinned VM,
libvirtd reset CPU affinity to all enabled in host CPUs, if it runs in
custom cpuset.
I can't reproduce this behavior with 4.5.0-10.el7_6.12 with the same
kernel version (Red Hat 7.7).
Libvirt runs in a custom cpuset 'libvirt', where the number of
available cpus is restricted to 0,2,4,6,8.
And this 'libvirt' cpuset is created in a system with total cpus
number: 40 (all cpus are enabled in BIOS) and I have '0-39' range in
/sys/fs/cgroup/cpuset/cpuset.cpus.
When a VM with pinned vcpus is launched, VM XML config is in attachement:
<vcpu placement='static'>2</vcpu>
<vcpupin vcpu='0' cpuset='4'/>
<vcpupin vcpu='1' cpuset='6'/>
I have the following error:
# virsh create /tmp/vm1_2vms-2cores_host1.xml
error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml
error: Unable to write to
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus':
Permission denied
And in the debug log I can see, that
/sys/fs/cgroup/cpuset/machine.slice was created with a proper cpuset
list: 0,2,4,6,8.
This list at the beginning was successfully inherited and set in
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus.
But finally libvirtd tries to reset inherited cpus and set there: "0-39"
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 :
Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 :
Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 :
Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1049 :
Make group /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug :
virCgroupCpuSetInherit:989 : Setting up inheritance
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope ->
/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus
2019-08-28 15:11:03.357+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.357+0000: 25536: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus'
to '0,2,4,6,8'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.mems'
to '0-1'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.memory_migrate'
to '1'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller memory
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller devices
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller freezer
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller blkio
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller net_cls
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller perf_event
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1055 :
Not creating systemd controller group
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1126 :
Done making controllers for group
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus'
to '0-39'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: error : virCgroupSetValueStr:806
: Unable to write to
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus':
Permission denied
The same log snippet with 4.5.0-10.el7_6.12 version, everything works well:
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 :
Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 :
Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 :
Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1049 :
Make group /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 :
Make controller
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.838+0000: 26937: debug :
virCgroupCpuSetInherit:989 : Setting up inheritance
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope ->
/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.cpus'
to '0,2,4,6,8'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems'
to '0-1'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832
: Get value
/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug :
virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.memory_migrate'
to '1'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller memory
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller devices
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller freezer
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller blkio
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller net_cls
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 :
Skipping unmounted controller perf_event
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1055 :
Not creating systemd controller group
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1126 :
Done making controllers for group
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems'
to '0'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796
: Set value
'/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/tasks'
to '27045'
Would someone give me a hint, please, if we need to change settings
lied to NUMA or vCPUs in VM XML config according to last changed in
libvirtd ?
Please, find a bug report about this issue here:
https://bugzilla.redhat.com/show_bug.cgi?id=1746517
I suppose, that this may be an impact of one of the following patches
applied to 4.5.0-10.el7_6.12 source code version:
libvirt-qemu-Rework-setting-process-affinity.patch
libvirt-qemu-Fix-qemuProcessInitCpuAffinity.patch
libvirt-qemu-Fix-NULL-pointer-access-in-qemuProcessInitCpuAffinity.patch
libvirt-qemu-Fix-leak-in-qemuProcessInitCpuAffinity.patch
How reproducible: Always
Steps to Reproduce:
1. create a cpuset with reduced cpus list and launch libvirtd in it
2. prepare VM XML with pinned vcpus and <numatune> setting
3. virsh create vm.xml
Actual results:
error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml
error: Unable to write to
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus':
Permission denied
Expected results:
should work as with a previous version 4.5.0-10.el7_6.12.
Additional info:
# ps auxf | grep libvirt
root 24571 2.3 0.0 1895564 28052 ? Ssl 16:52 0:00
/usr/sbin/libvirtd
# grep Cpus_allowed /proc/24571/task/*/status
/proc/24571/task/24571/status:Cpus_allowed: 00,00000554
/proc/24571/task/24571/status:Cpus_allowed_list: 2,4,6,8,10
/proc/24571/task/24572/status:Cpus_allowed: 00,00000554
/proc/24571/task/24572/status:Cpus_allowed_list: 2,4,6,8,10
/proc/24571/task/24573/status:Cpus_allowed: 00,00000554
/proc/24571/task/24573/status:Cpus_allowed_list: 2,4,6,8,10
/proc/24571/task/24574/status:Cpus_allowed: 00,00000554
/proc/24571/task/24574/status:Cpus_allowed_list: 2,4,6,8,10
/proc/24571/task/24575/status:Cpus_allowed: 00,00000554
..
for all threads
Processor: "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz"
Sockets: 2, Cores: 20, HyperThreads: 40
socket 0 socket 1
+---------------------+ +---------------------+
| c0 c1 | | c0 c1 |
| +-------+ +-------+ | | +-------+ +-------+ |
| | 0| 20| | 2| 22| | | | 1| 21| | 3| 23| |
| +-------+ +-------+ | | +-------+ +-------+ |
| c2 c3 | | c2 c3 |
| +-------+ +-------+ | | +-------+ +-------+ |
| | 4| 24| | 6| 26| | | | 5| 25| | 7| 27| |
| +-------+ +-------+ | | +-------+ +-------+ |
| c4 c8 | | c4 c8 |
| +-------+ +-------+ | | +-------+ +-------+ |
| | 8| 28| | 10| 30| | | | 9| 29| | 11| 31| |
| +-------+ +-------+ | | +-------+ +-------+ |
| c9 c10 | | c9 c10 |
| +-------+ +-------+ | | +-------+ +-------+ |
| | 12| 32| | 14| 34| | | | 13| 33| | 15| 35| |
| +-------+ +-------+ | | +-------+ +-------+ |
| c11 c12 | | c11 c12 |
| +-------+ +-------+ | | +-------+ +-------+ |
| | 16| 36| | 18| 38| | | | 17| 37| | 19| 39| |
| +-------+ +-------+ | | +-------+ +-------+ |
+---------------------+ +---------------------+
# cat /sys/fs/cgroup/cpuset/cpuset.cpus
0-39
# cat /sys/fs/cgroup/cpuset/libvirt/cpuset.cpus
2,4,6,8,10
# cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus
0,2,4,6,8