I have created the HELM Chart for deploying redis with sentinel, and access it with service using port-forwarding. After forwarding the port to 6379 I can set or get in redis-cli. However,
When I delete the pod from the statefulset, I could not able to access redis-cli anymore and get prompt Eg. :
127.0.0.1:6379> get b
Error: Server closed the connection
Basically sentinel should communicate and change the master pod and connection should be still there. but in my case it’s not working properly.
2
Answers
There’s so many issues with the redis chart but at the same time there are many problems with port forwarding so problem seems to be double complicated.
Many who worked with port forwarding noticed that it is visibly slower than connecting to a pod via a service, and the command just stop after a couple of minutes. So I advice you to not use it for production system.
You can also use proxy instead of port forwarding, example:
If you use kubectl proxy it is possible to reach different pods, as expected.
Port forwarding is originally designed to attach to one pod for debugging and other purpose. There was introduced new feature which help with service discovery so you don’t need to lookup the pod name first if you don’t care which one you attach to if you have multiple qualifying ones. Port forwarding enables http connections, goal is that application level load balancing does not apply and multiple active endpoints are not supported. There is a separate feature request for re-attach if the active pod terminates.
You can also default build port forwarding adding following lines into pod configuration file:
Or you can simple after pod recovery configure port forwarding once again.
More info you can find here: kubectl-port-forward.
If its a bitnami based redis sentinel cluster, it is possible that the sentinel election is not happening because of race conditions in the startup script. I came across this comment: https://github.com/bitnami/charts/issues/6165#issuecomment-852905670 and applying this fix: https://github.com/bitnami/charts/issues/6165#issuecomment-857859224 helped in my case.