Hello All,

Since 4.5.0-23.el7 version (Red Hat 7.7), when I launch pinned VM, libvirtd reset CPU affinity to all enabled in host CPUs, if it runs in custom cpuset.  
I can't reproduce this behavior with 4.5.0-10.el7_6.12 with the same kernel version (Red Hat 7.7).

Libvirt runs in a custom cpuset 'libvirt', where the number of available cpus is restricted to 0,2,4,6,8. 
And this 'libvirt' cpuset is created in a system with total cpus number: 40 (all cpus are enabled in BIOS) and I have '0-39' range in /sys/fs/cgroup/cpuset/cpuset.cpus. 

When a VM with pinned vcpus is launched, VM XML config is in attachement:

  <vcpu placement='static'>2</vcpu>
  <vcpupin vcpu='0' cpuset='4'/>
  <vcpupin vcpu='1' cpuset='6'/>

I have the following error:

# virsh create /tmp/vm1_2vms-2cores_host1.xml 
error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied

And in the debug log I can see, that /sys/fs/cgroup/cpuset/machine.slice was created with a proper cpuset list: 0,2,4,6,8. 
This list at the beginning was successfully inherited and set in /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus. 
But finally libvirtd tries to reset inherited cpus and set there: "0-39"

2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope -> /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus
2019-08-28 15:11:03.357+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus' to '0,2,4,6,8'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.mems' to '0-1'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.memory_migrate' to '1'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1126 : Done making controllers for group
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus' to '0-39'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: error : virCgroupSetValueStr:806 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied

The same log snippet with 4.5.0-10.el7_6.12 version, everything works well:

2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope -> /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.cpus' to '0,2,4,6,8'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems' to '0-1'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.memory_migrate' to '1'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1126 : Done making controllers for group
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems' to '0'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/tasks' to '27045'

Would someone give me a hint, please, if we need to change settings lied to NUMA or vCPUs in VM XML config according to last changed in libvirtd ?

Please, find a bug report about this issue here: https://bugzilla.redhat.com/show_bug.cgi?id=1746517

I suppose, that this may be an impact of one of the following patches applied to 4.5.0-10.el7_6.12 source code version:

libvirt-qemu-Rework-setting-process-affinity.patch
libvirt-qemu-Fix-qemuProcessInitCpuAffinity.patch
libvirt-qemu-Fix-NULL-pointer-access-in-qemuProcessInitCpuAffinity.patch
libvirt-qemu-Fix-leak-in-qemuProcessInitCpuAffinity.patch

How reproducible: Always

Steps to Reproduce:
1. create a cpuset with reduced cpus list and launch libvirtd in it
2. prepare VM XML with pinned vcpus and <numatune> setting
3. virsh create vm.xml

Actual results:

error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied

Expected results:
should work as with a previous version 4.5.0-10.el7_6.12.

Additional info:

# ps auxf | grep libvirt
root     24571  2.3  0.0 1895564 28052 ?       Ssl  16:52   0:00 /usr/sbin/libvirtd

# grep Cpus_allowed /proc/24571/task/*/status
/proc/24571/task/24571/status:Cpus_allowed:     00,00000554
/proc/24571/task/24571/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24572/status:Cpus_allowed:     00,00000554
/proc/24571/task/24572/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24573/status:Cpus_allowed:     00,00000554
/proc/24571/task/24573/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24574/status:Cpus_allowed:     00,00000554
/proc/24571/task/24574/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24575/status:Cpus_allowed:     00,00000554
..
for all threads

Processor: "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz"

        Sockets: 2, Cores: 20, HyperThreads: 40

         socket 0                 socket 1        
  +---------------------+  +---------------------+
  |        c0        c1 |  |        c0        c1 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | |  0| 20| |  2| 22| |  | |  1| 21| |  3| 23| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |        c2        c3 |  |        c2        c3 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | |  4| 24| |  6| 26| |  | |  5| 25| |  7| 27| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |        c4        c8 |  |        c4        c8 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | |  8| 28| | 10| 30| |  | |  9| 29| | 11| 31| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |        c9       c10 |  |        c9       c10 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | | 12| 32| | 14| 34| |  | | 13| 33| | 15| 35| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |       c11       c12 |  |       c11       c12 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | | 16| 36| | 18| 38| |  | | 17| 37| | 19| 39| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  +---------------------+  +---------------------+

# cat  /sys/fs/cgroup/cpuset/cpuset.cpus 
0-39

# cat  /sys/fs/cgroup/cpuset/libvirt/cpuset.cpus 
2,4,6,8,10

# cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 
0,2,4,6,8