[libvirt] increase daemon task limit
I recently received a report of a libvirt+Xen installation reaching the pids cgroup controller TasksMax limit kernel: [71282.213347] cgroup: fork rejected by pids controller in /system.slice/libvirtd.service The default setting of TaskMax is 512 on this system # cat /sys/fs/cgroup/pids/system.slice/libvirtd.service/pids.max 512 Depending on domain type and configuration, I've noticed between 5-7 tasks (IO threads, qemu process, etc.) are created when starting a domain. It doesn't take too many domains before the 512 limit is reached. LimitNOFILE was recently changed by commit 27cd763500 to support 4096 domains. Following similar logic, would it be ok to increase TasksMax to 32768? That would accommodate 4096 domains with 8 tasks each. TasksMax also supports the special value of "infinity", but that seems a bit aggressive to me. Regards, Jim
On Wed, Apr 19, 2017 at 09:03:55AM -0600, Jim Fehlig wrote:
I recently received a report of a libvirt+Xen installation reaching the pids cgroup controller TasksMax limit
kernel: [71282.213347] cgroup: fork rejected by pids controller in /system.slice/libvirtd.service
The default setting of TaskMax is 512 on this system
# cat /sys/fs/cgroup/pids/system.slice/libvirtd.service/pids.max 512
Depending on domain type and configuration, I've noticed between 5-7 tasks (IO threads, qemu process, etc.) are created when starting a domain. It doesn't take too many domains before the 512 limit is reached.
LimitNOFILE was recently changed by commit 27cd763500 to support 4096 domains. Following similar logic, would it be ok to increase TasksMax to 32768? That would accommodate 4096 domains with 8 tasks each. TasksMax also supports the special value of "infinity", but that seems a bit aggressive to me.
KVM uses 1 thread per vCPU, so 8 pids is probably somewhat on the low side if considering high vcpu counts. I think we could go for something like 128k pids which allows for closer to 32 vcpus per guest Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 04/19/2017 09:16 AM, Daniel P. Berrange wrote:
On Wed, Apr 19, 2017 at 09:03:55AM -0600, Jim Fehlig wrote:
I recently received a report of a libvirt+Xen installation reaching the pids cgroup controller TasksMax limit
kernel: [71282.213347] cgroup: fork rejected by pids controller in /system.slice/libvirtd.service
The default setting of TaskMax is 512 on this system
# cat /sys/fs/cgroup/pids/system.slice/libvirtd.service/pids.max 512
Depending on domain type and configuration, I've noticed between 5-7 tasks (IO threads, qemu process, etc.) are created when starting a domain. It doesn't take too many domains before the 512 limit is reached.
LimitNOFILE was recently changed by commit 27cd763500 to support 4096 domains. Following similar logic, would it be ok to increase TasksMax to 32768? That would accommodate 4096 domains with 8 tasks each. TasksMax also supports the special value of "infinity", but that seems a bit aggressive to me.
KVM uses 1 thread per vCPU, so 8 pids is probably somewhat on the low side if considering high vcpu counts. I think we could go for something like 128k pids which allows for closer to 32 vcpus per guest
It looks like the vcpu threads are under /sys/fs/cgroup/pids/machine.slice/machine-qemu<name>kvm.scope/, where pids.max is already 16k. I don't think these would be constrained by attributes under /sys/fs/cgroup/pids/system.slice/libvirtd.service. In fact, I don't see any additional tasks under /sys/fs/cgroup/pids/system.slice/libvirtd.service/tasks when starting a KVM guest. Since the libxl library runs in libvirtd process, any threads/processes it creates are accounted for in /sys/fs/cgroup/pids/system.slice/libvirtd.service. We are probably safe with the more conservative 32k. Regards, Jim
On Wed, Apr 19, 2017 at 01:11:41PM -0600, Jim Fehlig wrote:
On 04/19/2017 09:16 AM, Daniel P. Berrange wrote:
On Wed, Apr 19, 2017 at 09:03:55AM -0600, Jim Fehlig wrote:
I recently received a report of a libvirt+Xen installation reaching the pids cgroup controller TasksMax limit
kernel: [71282.213347] cgroup: fork rejected by pids controller in /system.slice/libvirtd.service
The default setting of TaskMax is 512 on this system
# cat /sys/fs/cgroup/pids/system.slice/libvirtd.service/pids.max 512
Depending on domain type and configuration, I've noticed between 5-7 tasks (IO threads, qemu process, etc.) are created when starting a domain. It doesn't take too many domains before the 512 limit is reached.
LimitNOFILE was recently changed by commit 27cd763500 to support 4096 domains. Following similar logic, would it be ok to increase TasksMax to 32768? That would accommodate 4096 domains with 8 tasks each. TasksMax also supports the special value of "infinity", but that seems a bit aggressive to me.
KVM uses 1 thread per vCPU, so 8 pids is probably somewhat on the low side if considering high vcpu counts. I think we could go for something like 128k pids which allows for closer to 32 vcpus per guest
It looks like the vcpu threads are under /sys/fs/cgroup/pids/machine.slice/machine-qemu<name>kvm.scope/, where pids.max is already 16k. I don't think these would be constrained by attributes under /sys/fs/cgroup/pids/system.slice/libvirtd.service. In fact, I don't see any additional tasks under /sys/fs/cgroup/pids/system.slice/libvirtd.service/tasks when starting a KVM guest.
Since the libxl library runs in libvirtd process, any threads/processes it creates are accounted for in /sys/fs/cgroup/pids/system.slice/libvirtd.service. We are probably safe with the more conservative 32k.
Oh I see, I didn't realize it was a xen specific issue. I agree with your suggestion Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (2)
-
Daniel P. Berrange -
Jim Fehlig