skip to Main Content

Small question regarding Redis deployed in AWS (not AWS Elastic Cache) and an issue connecting to it.

Here is the setup of the Redis deployed in AWS: (pasting only the Kubernetes StatefulSet and Service)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: redis
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
        - name: config
          image: redis:7.0.5-alpine
          command: [ "sh", "-c" ]
          args:
            - |
              cp /tmp/redis/redis.conf /etc/redis/redis.conf
              
              echo "finding master..."
              MASTER_FDQN=`hostname  -f | sed -e 's/redis-[0-9]./redis-0./'`
              if [ "$(redis-cli -h sentinel -p 5000 ping)" != "PONG" ]; then
                echo "master not found, defaulting to redis-0"
              
                if [ "$(hostname)" = "redis-0" ]; then
                  echo "this is redis-0, not updating config..."
                else
                  echo "updating redis.conf..."
                  echo "slaveof $MASTER_FDQN 6379" >> /etc/redis/redis.conf
                fi
              else
                echo "sentinel found, finding master"
                MASTER="$(redis-cli -h sentinel -p 5000 sentinel get-master-addr-by-name mymaster | grep -E '(^redis-d{1,})|([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})')"
                echo "master found : $MASTER, updating redis.conf"
                echo "slaveof $MASTER 6379" >> /etc/redis/redis.conf
              fi
          volumeMounts:
            - name: redis-config
              mountPath: /etc/redis/
            - name: config
              mountPath: /tmp/redis/
      containers:
        - name: redis
          image: redis:7.0.5-alpine
          command: ["redis-server"]
          args: ["/etc/redis/redis.conf"]
          ports:
            - containerPort: 6379
              name: redis
          volumeMounts:
            - name: data
              mountPath: /data
            - name: redis-config
              mountPath: /etc/redis/
      volumes:
        - name: redis-config
          emptyDir: {}
        - name: config
          configMap:
            name: redis-config
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: nfs-1
        resources:
          requests:
            storage: 50Mi
---
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  ports:
    - port: 6379
      targetPort: 6379
      name: redis
  selector:
    app: redis
  type: LoadBalancer

The pods are healthy, I can exec into it and perform operations fine. Here is the get all:

NAME          READY   STATUS    RESTARTS   AGE
pod/redis-0   1/1     Running   0          22h
pod/redis-1   1/1     Running   0          22h
pod/redis-2   1/1     Running   0          22h

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/redis   LoadBalancer   192.168.45.55   10.51.5.2     6379:30315/TCP   26h

NAME                     READY   AGE
statefulset.apps/redis   3/3     22h

Here is the describe of the service:

Name:                     redis
Namespace:                Namespace
Labels:                   <none>
Annotations:              <none>
Selector:                 app=redis
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       192.168.22.33
IPs:                      192.168.22.33
LoadBalancer Ingress:     10.51.5.2
Port:                     redis  6379/TCP
TargetPort:               6379/TCP
NodePort:                 redis  30315/TCP
Endpoints:                192.xxx:6379,192.xxx:6379,192.xxx:6379
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age                From                Message
  ----    ------        ----               ----                -------
  Normal  IPAllocated   68s                metallb-controller  Assigned IP ["10.51.5.2"]
  Normal  nodeAssigned  58s (x5 over 66s)  metallb-speaker     announcing from node "someaddress.com" with protocol "bgp"
  Normal  nodeAssigned  58s (x5 over 66s)  metallb-speaker     announcing from node "someaddress.com" with protocol "bgp"

I then try to connect to it, i.e. inserting some data with a very straightforward Spring Boot application. The application has no business logic, just trying to insert data.
Here are the relevant parts:


@Configuration
public class RedisConfiguration {

    @Bean
    public ReactiveRedisConnectionFactory reactiveRedisConnectionFactory() {
        return new LettuceConnectionFactory("10.51.5.2", 30315);
    }

@Repository
public class RedisRepository {

    private final ReactiveRedisOperations<String, String> reactiveRedisOperations;

    public RedisRepository(ReactiveRedisOperations<String, String> reactiveRedisOperations) {
        this.reactiveRedisOperations = reactiveRedisOperations;
    }

    public Mono<RedisPojo> save(RedisPojo redisPojo) {
        return reactiveRedisOperations.opsForValue().set(redisPojo.getInput(), redisPojo.getOutput()).map(__ -> redisPojo);
    }

Each time I am trying to write the data, I am getting this exception:

2022-12-02T20:20:08.015+08:00 ERROR 1184 --- [ctor-http-nio-3] a.w.r.e.AbstractErrorWebExceptionHandler : [8f16a752-1]  500 Server Error for HTTP POST "/save"

org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.translateException(LettuceConnectionFactory.java:1602) ~[spring-data-redis-3.0.0.jar:3.0.0]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    *__checkpoint ⇢ Handler com.redis.controller.RedisController#test(RedisRequest) [DispatcherHandler]
    *__checkpoint ⇢ HTTP POST "/save" [ExceptionHandlingWebHandler]
Original Stack Trace:
        at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.translateException(LettuceConnectionFactory.java:1602) ~[spring-data-redis-3.0.0.jar:3.0.0]

Caused by: io.lettuce.core.RedisConnectionException: Unable to connect to 10.51.5.2/<unresolved>:30315
    at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78) ~[lettuce-core-6.2.1.RELEASE.jar:6.2.1.RELEASE]
    at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56) ~[lettuce-core-6.2.1.RELEASE.jar:6.2.1.RELEASE]
    at io.lettuce.core.AbstractRedisClient.getConnection(AbstractRedisClient.java:350) ~[lettuce-core-6.2.1.RELEASE.jar:6.2.1.RELEASE]
    at io.lettuce.core.RedisClient.connect(RedisClient.java:216) ~[lettuce-core-6.2.1.RELEASE.jar:6.2.1.RELEASE]

Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /10.51.5.2:30315
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261) ~[netty-transport-4.1.85.Final.jar:4.1.85.Final]
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.85.Final.jar:4.1.85.Final]

This is particularly puzzling, because I am quite sure the code of the Spring Boot app is working. When I change the IP of return new LettuceConnectionFactory("10.51.5.2", 30315);: to

  1. a regular Redis on my laptop ("localhost", 6379),
  2. a dockerized Redis on my laptop,
  3. a dockerized Redis on prem, all are working fine.

Therefore, I am quite puzzled what did I do wrong with the setup of this Redis in AWS.
What should I do in order to connect to it properly.
May I get some help please?

Thank you

2

Answers


  1. By default, Redis binds itself to the IP addresses 127.0.0.1 and ::1 and does not accept connections against non-local interfaces. Chances are high that this is your main issue and you may want to review your redis.conf file to bind Redis to the interface you need or to the generic * -::*, as explained in the comments of the config file itself (which I have linked above).

    With that being said, Redis also does not accept connections on non-local interfaces if the default user has no password – a security layer named Protected mode. Thus you should either give your default user a password or disable protected mode in your redis.conf file.

    Not sure if this applies to your case but, as a side note, I would suggest to always avoid exposing Redis to the Internet.

    Login or Signup to reply.
  2. You are mixing 2 things.
    To enable this service for pods in different namespaces you do not need external load balancer, you can just try to use redis.namespace-name:6379 dns name and it will just work. Such dns is there for every service you create (but works only inside kubernetes)
    Kubernetes will make sure that your traffic will be routed to proper pods (assuming there is more than one).

    If you want to expose redis from outside of kubernetes then you need to make sure there is connectivity from the outside and then you need network load balancer that will forward traffic to your kubernetes service (in your case node port, so you need NLB with eks worker nodes: 30315 as a targets)

    If your worker nodes have public IP and their SecurityGroups allow connecting to them directly, you could try to connect to worker node’s IP directly just to test things out (without LB).

    And regardless off yout setup you can always create proxy via kubectl

    kubectl port-forward -n redisNS svc/redis 6379:6379
    

    and connect from spring boot app to localhost:6379

    How do you want to connect from app to redis in a final setup?

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search