skip to Main Content

I am trying to configure a redis statefulset on a specific node_pool, but when I apply the manifest, I get the following error.

0/4 nodes are available: 1 node(s) had volume node affinity conflict, 3 node(s) didn't match Pod's node affinity/selector.

My GKE is Regional and I create a nodepool with one replica in one zone, because it is sufficient for the redis work.

The node_pool is in a single zone. My idea is that the 3 replicas live inside that particular node.

How can I configure that the 3 PVCs are in the same node_pool?

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: redis
spec:
  serviceName: redis
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
      - name: config
        image: redis:6.2.3-alpine
        command: [ "sh", "-c" ]
        args:
          - |
            cp /tmp/redis/redis.conf /etc/redis/redis.conf
            
            echo "finding master..."
            MASTER_FDQN=`hostname  -f | sed -e 's/redis-[0-9]./redis-0./'`
            if [ "$(redis-cli -h sentinel -p 5000 ping)" != "PONG" ]; then
              echo "master not found, defaulting to redis-0"
              if [ "$(hostname)" == "redis-0" ]; then
                echo "this is redis-0, not updating config..."
              else
                echo "updating redis.conf..."
                echo "slaveof $MASTER_FDQN 6379" >> /etc/redis/redis.conf
              fi
            else
              echo "sentinel found, finding master"
              MASTER="$(redis-cli -h sentinel -p 5000 sentinel get-master-addr-by-name mymaster | grep -E '(^redis-d{1,})|([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})')"
              echo "master found : $MASTER, updating redis.conf"
              echo "slaveof $MASTER 6379" >> /etc/redis/redis.conf
            fi
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis/
        - name: config
          mountPath: /tmp/redis/
      containers:
      - name: redis
        image: redis:6.2.3-alpine
        command: ["redis-server"]
        args: ["/etc/redis/redis.conf"]
        ports:
        - containerPort: 6379
          name: redis
        volumeMounts:
        - name: data
          mountPath: /data
        - name: redis-config
          mountPath: /etc/redis/
      volumes:
      - name: redis-config
        emptyDir: {}
      - name: config
        configMap:
          name: redis-config
      nodeSelector:
        cloud.google.com/gke-nodepool: redis-pool
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "standard"
      resources:
        requests:
          storage: 50Mi
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: redis
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: redis
  selector:
    app: redis

2

Answers


  1. Chosen as BEST ANSWER

    UPDATE: I found a solution, i create a StorageClass with a "matchLabelExpressions" matching a specific zone, so I define to use that StorageClass on my StafefulSet manifest matching the StorageClass name to "data"

    Statefulset

    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: redis
      namespace: redis
    spec:
      serviceName: redis
      replicas: 3
      selector:
        matchLabels:
          app: redis
      template:
        metadata:
          labels:
            app: redis
        spec:
          initContainers:
          - name: config
            image: redis:6.2.3-alpine
            command: [ "sh", "-c" ]
            args:
              - |
                cp /tmp/redis/redis.conf /etc/redis/redis.conf
                
                echo "finding master..."
                MASTER_FDQN=`hostname  -f | sed -e 's/redis-[0-9]./redis-0./'`
                if [ "$(redis-cli -h sentinel -p 5000 ping)" != "PONG" ]; then
                  echo "master not found, defaulting to redis-0"
                  if [ "$(hostname)" == "redis-0" ]; then
                    echo "this is redis-0, not updating config..."
                  else
                    echo "updating redis.conf..."
                    echo "slaveof $MASTER_FDQN 6379" >> /etc/redis/redis.conf
                  fi
                else
                  echo "sentinel found, finding master"
                  MASTER="$(redis-cli -h sentinel -p 5000 sentinel get-master-addr-by-name mymaster | grep -E '(^redis-d{1,})|([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})')"
                  echo "master found : $MASTER, updating redis.conf"
                  echo "slaveof $MASTER 6379" >> /etc/redis/redis.conf
                fi
            volumeMounts:
            - name: redis-config
              mountPath: /etc/redis/
            - name: config
              mountPath: /tmp/redis/
          containers:
          - name: redis
            image: redis:6.2.3-alpine
            command: ["redis-server"]
            args: ["/etc/redis/redis.conf"]
            ports:
            - containerPort: 6379
              name: redis
            volumeMounts:
            - name: data
              mountPath: /data
            - name: redis-config
              mountPath: /etc/redis/
          volumes:
          - name: redis-config
            emptyDir: {}
          - name: config
            configMap:
              name: redis-config
          nodeSelector:
            cloud.google.com/gke-nodepool: redis-pool
      volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: "data"
          resources:
            requests:
              storage: 50Mi
    

    StorageClass

    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: data
      namespace: redis
    provisioner: pd.csi.storage.gke.io
    parameters:
      type: pd-standard
      replication-type: none
    volumeBindingMode: WaitForFirstConsumer
    allowedTopologies:
    - matchLabelExpressions:
      - key: topology.gke.io/zone
        values:
        - europe-west3-a
    
    

  2. Can you ensure that the nodes in your node_pool have the label redis-pool? From your requirements, you should only have 1 node with this label, which will allow all 3 replicas to be scheduled to this node.

    And it is not technically correct to say PVCs in the same node_pool. You are using the storageClass: default in your PVC template, this will cause your PersistentVolumeClaims to be backed by Google Persistent Disks (GPDs). Which are separate from your node_pools (compute resource) and should not be considered part of the compute infrastructure.

    They belong to the storage part of your GCloud. The GKE however handles the connection between the storage and compute transparently to you, so it doesn’t feel like you’re using the GPDs, but it is important to understand this distinction.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search