Nginx container not restarting after shutdown in Kubernetes

Okay! After much intensive trial and error, this is what ultimately worked for us.

1. We set up a liveliness & readiness probe in our Front End/upstream nginx containers and in the deployment.

nginx.conf

# default server block
location /healthz {
   auth_basic off;
   allow all;
   return 200;
}

deployment.yaml

#spec.containers
livenessProbe:
  failureThreshold: 1
  httpGet:
    path: /healthz
    port: 80
readinessProbe:
  failureThreshold: 3
  httpGet:
    path: /healthz
    port: 80

This ensured that, without a doubt, when our nginx containers failed, for whatever reason, they would be restarted.

2. We set up a smooth shutdown for the nginx containers

deployment.yaml

#spec.containers
lifecycle:
  preStop:
    exec:
      command: ["/bin/sh","-c","sleep 3; nginx -s quit; while pgrep -x nginx; do sleep 1; done"]

This ensured that when the restarts were happening, the nginx containers actually shutdown without losing connections (though to be fair this was rarer than not).

3. Minimum CPU and Memory Requests

With the above two changes in place, it felt like we were in a good place. Until, we scaled out to 3 replicas per Front End deployment. After just one night, seemingly randomly chosen, a few of the replicas across a couple of the FE deployments ( we had set up several for testing, dev-1 through dev-20) had restarted over 800 times! Insanity!

We didn't know what could be causing this. But on a whim, we ended up adding some minimum CPU and Memory requests, and that ended up solving the restart/failure cycle.

# spec.containers
resources:
  requests:
    cpu: 0.001
    memory: 4M

Final Notes

As you can see from the above, we really ended up adding a lot of configuration and work to our existing deployment set up. But none of it was Ingress related, it was all our upstream nginx containers.

The liveliness probe is pretty strict because once the containers began acting up it was game over for that container. We had not witnessed the containers fix themselves, so once the liveliness probe caught that the health check was failing, we were fine with forcing a restart.

The preStop command is roughly what was recommended as the default for nginx in the kubernetes documentation ON preStop commands as shown here. The sleep 3 was found in another stackOverflow question, while pgrep was used in this medium article. It's definitely a cobbled together stop command, but it seems to be working for us.

Finally, the minimum CPU request and Memory requests we came upon were based on the metrics for all of our FE Deployments. 0.001 CPU was the the mean usage during higher load, and 4M was the mean usage for memory during higher load. Figured it is best to hedge out bets here and use slightly more CPU/Memory than we need (though obviously this CPU usage is tiny).

I am not entirely sure why setting up the requests is what finally sorted out the restart issue. If I had to take a guess, giving the requests allowed kubernetes to find better nodes to put these pods on. Perhaps the pods that were constantly failing were crammed into the very limits of one node or another and so any actual usage caused failure.

I hope this helps someone in the future, or perhaps this was a monster of our own making and everyone already knows to do all of the above.

Good luck to whoever finds this!