Azure Key Vault integration with AKS works for nginx tutorial Pod, but not actual project deployment

cjones
February 22, 2021
176 views
0 votes
2 Answers

Per the title, I have the integration working following the documentation.

I can deploy the nginx.yaml and after about 70 seconds I can print out secrets with:

kubectl exec -it nginx -- cat /mnt/secrets-store/secret1

Now I’m trying to apply it to a PostgreSQL deployment for testing and I get the following from the Pod description:

  Warning  FailedMount             3s    kubelet                  MountVolume.SetUp failed for volume "secrets-store01-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod staging/postgres-deployment-staging-69965ff767-8hmww, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to get keyvault client: failed to get key vault token: nmi response failed with status code: 404, err: <nil>

And from the nmi logs:

E0221 22:54:32.037357       1 server.go:234] failed to get identities, error: getting assigned identities for pod staging/postgres-deployment-staging-69965ff767-8hmww in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0221 22:54:32.037409       1 server.go:192] status (404) took 80003389208 ns for req.method=GET reg.path=/host/token/ req.remote=127.0.0.1

Not sure why since I basically copied the settings from the nignx.yaml into the postgres.yaml. Here they are:

# nginx.yaml
kind: Pod
apiVersion: v1
metadata:
  name: nginx
  namespace: staging
  labels:
    aadpodidbinding: aks-akv-identity-binding-selector
spec:
  containers:
    - name: nginx
      image: nginx
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: /mnt/secrets-store
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: aks-akv-secret-provider

# postgres.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-deployment-staging
  namespace: staging
  labels:
    aadpodidbinding: aks-akv-identity-binding-selector
spec:
  replicas: 1
  selector:
    matchLabels:
      component: postgres
  template:
    metadata:
      labels:
        component: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:13-alpine
          ports:
            - containerPort: 5432
          volumeMounts:
          - name: secrets-store01-inline
            mountPath: /mnt/secrets-store
            readOnly: true
          - name: postgres-storage-staging
            mountPath: /var/postgresql
      volumes:
        - name: secrets-store01-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: aks-akv-secret-provider
        - name: postgres-storage-staging
          persistentVolumeClaim:
            claimName: postgres-storage-staging

---
apiVersion: v1
kind: Service
metadata:
  name: postgres-cluster-ip-service-staging
  namespace: staging
spec:
  type: ClusterIP
  selector:
    component: postgres
  ports:
    - port: 5432
      targetPort: 5432

Suggestions for what the issue is here?

Answers

Chosen as BEST ANSWER

Oversight on my part... the aadpodidbinding should be in the template: per:

https://azure.github.io/aad-pod-identity/docs/best-practices/#deploymenthttpskubernetesiodocsconceptsworkloadscontrollersdeployment

The resulting YAML should be:

# postgres.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-deployment-production
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      component: postgres
  template:
    metadata:
      labels:
        component: postgres
        aadpodidbinding: aks-akv-identity-binding-selector
    spec:
      containers:
        - name: postgres
          image: postgres:13-alpine
          ports:
            - containerPort: 5432
          env: 
            - name: POSTGRES_DB_FILE
              value: /mnt/secrets-store/DEV-PGDATABASE
            - name: POSTGRES_USER_FILE
              value: /mnt/secrets-store/DEV-PGUSER
            - name: POSTGRES_PASSWORD_FILE
              value: /mnt/secrets-store/DEV-PGPASSWORD
            - name: POSTGRES_INITDB_ARGS
              value: "-A md5"
            - name: PGDATA
              value: /var/postgresql/data
          volumeMounts:
          - name: secrets-store01-inline
            mountPath: /mnt/secrets-store
            readOnly: true
          - name: postgres-storage-production
            mountPath: /var/postgresql
      volumes:
        - name: secrets-store01-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: aks-akv-secret-provider
        - name: postgres-storage-production
          persistentVolumeClaim:
            claimName: postgres-storage-production
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-cluster-ip-service-production
  namespace: production
spec:
  type: ClusterIP
  selector:
    component: postgres
  ports:
    - port: 5432
      targetPort: 5432

(Edit)

Adding template in spec will resolve the issue, use label "aadpodidbinding: "your azure pod identity selector" in the template labels section in deployment.yaml file

sample deployment file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
        aadpodidbinding: azure-pod-identity-binding-selector
    spec:
      containers:
        - name: nginx
          image: nginx
          env:
          - name: SECRET
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: key
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: dev-1spc

Please signup or login to give your own answer.

Click here to cancel reply.