nginx-ingress unable to reload worker processes

What happened?

Just deploy a fresh k8s cluster with crio, deploy ingress-nginx and try to load ingresses. Nginx throws errors because it cannot respawn its worker processes:

2022/04/05 09:27:49 [alert] 56#56: sendmsg() failed (9: Bad file descriptor)
2022/04/05 09:27:49 [alert] 56#56: sendmsg() failed (9: Bad file descriptor)
2022/04/05 09:27:49 [alert] 1415#1415: pthread_create() failed (11: Resource temporarily unavailable)
2022/04/05 09:27:49 [alert] 1411#1411: pthread_create() failed (11: Resource temporarily unavailable)
2022/04/05 09:27:49 [alert] 1431#1431: pthread_create() failed (11: Resource temporarily unavailable)
2022/04/05 09:27:50 [alert] 56#56: worker process 1190 exited with fatal code 2 and cannot be respawned
2022/04/05 09:27:50 [alert] 56#56: worker process 1191 exited with fatal code 2 and cannot be respawned

If i just switch from crio to docker it works without any error - thats why i think the issue is related to crio and not to ingress-nginx.

The issue only happens if nginx has many worker processes. With testing i think the sweet spot is around 14- 18 worker processes.

We ran into the issue because by default, nginx-ingress has an auto setting for worker processes which spawns as many workers as cores are detected. If you have large systems with xxx cores the issue occures.

What did you expect to happen?

Nginx should work without any issues / should be able to spawn its worker processes.

How can we reproduce it (as minimally and precisely as possible)?

Yum install

yum install kubelet-1.20.15 kubeadm-1.20.15 kubectl-1.20.15 crio crictl  --disableexcludes=kubernetes

Create k8s cluster

kubeadm init --pod-network-cidr=192.168.0.0/16
kubectl apply -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml
kubectl apply -f https://projectcalico.docs.tigera.io/manifests/custom-resources.yam
kubectl taint nodes --all node-role.kubernetes.io/master-

Prepare configs

$ cat values.nginx-ingress-default.yaml 

controller:
  config:
    worker-processes: "40"

$ cat ingress.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test
spec:
  ingressClassName: nginx
  rules:
  - host: test.tld
    http:
      paths:
      - backend:
          service:
            name: backend
            port:
              number: 80
        path: /
        pathType: Prefix

Install nginx-ingress

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-default ingress-nginx/ingress-nginx --values values.nginx-ingress-default.yaml --version 4.0.18
kubectl create -f ingressl.yaml

Anything else we need to know?

If i set worker-processes to a lower value like 4x or 8x it works with crio.

If i set higher values like 14x i see that some worker processes are able to respawn and others fail with the error:

2022/04/05 09:18:59 [alert] 647#647: pthread_create() failed (11: Resource temporarily unavailable)
2022/04/05 09:18:59 [alert] 645#645: pthread_create() failed (11: Resource temporarily unavailable)
2022/04/05 09:19:00 [alert] 56#56: worker process 665 exited with fatal code 2 and cannot be respawned
2022/04/05 09:19:00 [alert] 56#56: worker process 698 exited with fatal code 2 and cannot be respawned

You can verify that from within the container if you run ps auxf and count the worker processes:

bash-5.1$ ps auxf
PID   USER     TIME  COMMAND
    1 www-data  0:00 /usr/bin/dumb-init -- /nginx-ingress-controller --publish-service=ix-system/ix-ingress-default-ingress-nginx-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=ix-system/ix-ingress-default-ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certif
    7 www-data  0:00 /nginx-ingress-controller --publish-service=ix-system/ix-ingress-default-ingress-nginx-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=ix-system/ix-ingress-default-ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certi
   57 www-data  0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /etc/nginx/nginx.conf
   61 www-data  0:00 nginx: worker process
   62 www-data  0:00 nginx: worker process
   63 www-data  0:00 nginx: cache manager process

If you set for example 20x worker processes via config file you should see 8x worker processes via ps auxf (after restarting the container / creating a new ingress object which triggers a realod) and 12x the error mesage from above; That means that 8x workeer processes are able to spawn and 12x are not (you see then 12x times the failed / exited error message).

Like i said before, if i remove crio and install docker everything works like it should.

Selinux is disabled. cgroups are managed via systemd (for both, crio and k8s).

CRI-O and Kubernetes version

$ crio --version
INFO[0000] Starting CRI-O, version: 1.20.7, git: 5d2fa3419f15655454edb94dfcc0993cb5a1521c(clean) 
crio version 1.20.7
Version:       1.20.7
GitCommit:     5d2fa3419f15655454edb94dfcc0993cb5a1521c
GitTreeState:  clean
BuildDate:     2022-03-15T18:16:57Z
GoVersion:     go1.16.12
Compiler:      gc
Platform:      linux/amd64
Linkmode:      dynamic

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.15", GitCommit:"8f1e5bf0b9729a899b8df86249b56e2c74aebc55", GitTreeState:"clean", BuildDate:"2022-01-19T17:27:39Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.15", GitCommit:"8f1e5bf0b9729a899b8df86249b56e2c74aebc55", GitTreeState:"clean", BuildDate:"2022-01-19T17:23:01Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

OS version

$ uname -a
Linux 86-187-116-62.customer-virt.eu 4.18.0-348.2.1.el8_5.x86_64 #1 SMP Mon Nov 15 09:17:08 EST 2021 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release 
AlmaLinux release 8.5 (Arctic Sphynx)

nginx-ingress unable to reload worker processes #5779