Re: [libvirt] increase daemon task limit

On 04/19/2017 09:16 AM, Daniel P. Berrange wrote: > On Wed, Apr 19, 2017 at 09:03:55AM -0600, Jim Fehlig wrote: > > I recently received a report of a libvirt+Xen installation reaching the pids > > cgroup controller TasksMax limit > > > > kernel: [71282.213347] cgroup: fork rejected by pids controller in > > /system.slice/libvirtd.service > > > > The default setting of TaskMax is 512 on this system > > > > # cat /sys/fs/cgroup/pids/system.slice/libvirtd.service/pids.max > > 512 > > > > Depending on domain type and configuration, I've noticed between 5-7 tasks > > (IO threads, qemu process, etc.) are created when starting a domain. It > > doesn't take too many domains before the 512 limit is reached. > > > > LimitNOFILE was recently changed by commit 27cd763500 to support 4096 > > domains. Following similar logic, would it be ok to increase TasksMax to > > 32768? That would accommodate 4096 domains with 8 tasks each. TasksMax also > > supports the special value of "infinity", but that seems a bit aggressive to > > me. > > KVM uses 1 thread per vCPU, so 8 pids is probably somewhat on the low > side if considering high vcpu counts. I think we could go for something > like 128k pids which allows for closer to 32 vcpus per guest It looks like the vcpu threads are under /sys/fs/cgroup/pids/machine.slice/machine-qemu<name>kvm.scope/, where pids.max is already 16k. I don't think these would be constrained by attributes under /sys/fs/cgroup/pids/system.slice/libvirtd.service. In fact, I don't see any additional tasks under /sys/fs/cgroup/pids/system.slice/libvirtd.service/tasks when starting a KVM guest. Since the libxl library runs in libvirtd process, any threads/processes it creates are accounted for in /sys/fs/cgroup/pids/system.slice/libvirtd.service. We are probably safe with the more conservative 32k.