April 21, 2019 03:58AM
Hi
We are trying to create ~4000 namespaces with ingress rules and we cannot reach above ~2000 namespaces…
Our problematic use case is:
1. We create k8s namespace with 1 pod (3 containers) and 1 ingress (3 rules per ingress).
2. As part of the creation of the namespace we run a few https queries and sync some data to the containers (GET & POST)
3. After creating ~2050 namespaces the NGINX ingress controller get into CrashLoopBackOf state and cannot restart anymore.

We use NGINX ingress controller with image 0.24.1 in AWS environment (mapped to nlb), we run a replica of 4 pods with 4cpu and 12Gi memory
We tried a few configuration changes but it seems we cannot find the root cause of the limitation.

In the logs of the nginx controller the problem start in the dynamic configuration with reason "no memory":
"[error] 50#50: *165 [lua] configuration.lua:182: call(): dynamic-configuration: error updating configuration: no memory
controller.go:220] Dynamic reconfiguration failed: unexpected error code: 400
controller.go:224] Unexpected failure reconfiguring NGINX"

It seems like we are missing some key configuration limit that causes k8s to kill the pod…

Appreciate your assistance!

Saggi Raiter.
Saggi.raiter@sap.com
Subject Author Posted

Ingress get into CrashLoopBackOf state

saggir April 21, 2019 03:58AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 56
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready