skip to Main Content

I have created the HELM Chart for deploying redis with sentinel, and access it with service using port-forwarding. After forwarding the port to 6379 I can set or get in redis-cli. However,
When I delete the pod from the statefulset, I could not able to access redis-cli anymore and get prompt Eg. :

127.0.0.1:6379> get b
Error: Server closed the connection

Basically sentinel should communicate and change the master pod and connection should be still there. but in my case it’s not working properly.

2

Answers


  1. There’s so many issues with the redis chart but at the same time there are many problems with port forwarding so problem seems to be double complicated.
    Many who worked with port forwarding noticed that it is visibly slower than connecting to a pod via a service, and the command just stop after a couple of minutes. So I advice you to not use it for production system.

    As we can read in official Kubernetes documentation Manual force
    deletion pod of
    StatefulSet should
    be undertaken with caution, as it has the potential to violate the at
    most one semantics inherent to StatefulSet. StatefulSets may be used
    to run distributed and clustered applications which have a need for a
    stable network identity and stable storage. These applications often
    have configuration which relies on an ensemble of a fixed number of
    members with fixed identities. Having multiple members with the same
    identity can be disastrous and may lead to data loss (e.g. split brain
    scenario in quorum-based systems)

    After deleting pod Pods’ ordinals, hostnames, SRV records, and A
    record names have not changed, but the IP addresses associated with
    the pod changed. In the cluster used for this tutorial, they have.
    This is why it is important not to configure other applications to
    connect to Pods in a StatefulSet by IP address.

    If you need to find and connect to the active members of a
    StatefulSet, you should query the CNAME of the Headless Service
    (nginx.default.svc.cluster.local). The SRV records associated with the
    CNAME will contain only the Pods in the StatefulSet that are Running
    and Ready.

    If your application already implements connection logic that tests for
    liveness and readiness, you can use the SRV records of the Pods (
    web-0.nginx.default.svc.cluster.local,
    web-1.nginx.default.svc.cluster.local), as they are stable, and your
    application will be able to discover the Pods’ addresses when they
    transition to Running and Ready.

    You can also use proxy instead of port forwarding, example:

    $ kubectl proxy  --port=6379
    

    If you use kubectl proxy it is possible to reach different pods, as expected.
    Port forwarding is originally designed to attach to one pod for debugging and other purpose. There was introduced new feature which help with service discovery so you don’t need to lookup the pod name first if you don’t care which one you attach to if you have multiple qualifying ones. Port forwarding enables http connections, goal is that application level load balancing does not apply and multiple active endpoints are not supported. There is a separate feature request for re-attach if the active pod terminates.

    You can also default build port forwarding adding following lines into pod configuration file:

    export YOURAPP_POD=$(kubectl get pods -n $NAMESPACE | grep your-app | awk '{print $1;}')
    kubectl port-forward -n $YOUR_NAMESPACE $YOURAPP_POD 8080
    

    Or you can simple after pod recovery configure port forwarding once again.

    More info you can find here: kubectl-port-forward.

    Login or Signup to reply.
  2. If its a bitnami based redis sentinel cluster, it is possible that the sentinel election is not happening because of race conditions in the startup script. I came across this comment: https://github.com/bitnami/charts/issues/6165#issuecomment-852905670 and applying this fix: https://github.com/bitnami/charts/issues/6165#issuecomment-857859224 helped in my case.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search