Cert-manager stopped renewing Let'S Encrypt certificates after upgrading to AKS 1.20.7 – Nginx

Our AKS cluster was configured to auto-renew Let’s Encrypt certificates through Ingress Cert-Manager annotation and this worked perfectly until we upgraded to AKS 1.20.7. This then stopped working and the certificates started to expire without them being renewed – I double-checked all changes to K8S and CertManager APIs and reviewed all YAMLs, but I’m not seeing anything obviously wrong. Would appreciate any pointers.

My understanding is that as long as I add the "cert-manager.io/cluster-issuer: letsencrypt-prod-p9v2" to my ingress – the whole renewal should happen automatically – this is not happening though.

> kubectl cert-manager version
util.Version{GitVersion:"v1.4.0", GitCommit:"5e2a6883c1202739902ac94b5f4884152b810925", GitTreeState:"clean", GoVersion:"go1.16.2", Compiler:"gc", Platform:"linux/amd64"}

AKS version: 1.20.7

cat shipit-ingress-p9v2.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod-p9v2
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 15m
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.org/client-max-body-size: 15m
  generation: 4
  name: shipit-ingress-p9v2
  namespace: supplier
  resourceVersion: "147087245"
  uid: 6751dbff-83b1-48a1-a467-e75cc843ee79
spec:
  rules:
  - host: xxx.westeurope.cloudapp.azure.com
    http:
      paths:
      - backend:
          service:
            name: planet9v2
            port:
              number: 8080
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - xxx.westeurope.cloudapp.azure.com
    secretName: tls-secret-p9v2
status:
  loadBalancer:
    ingress:
    - ip: 10.240.0.5

>>kubectl get clusterissuer -o yaml letsencrypt-prod-p9v2
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  annotations:
  creationTimestamp: "2020-05-29T13:31:10Z"
  generation: 2
  name: letsencrypt-prod-p9v2
  resourceVersion: "25493731"
  uid: 0e0e46f5-4cdf-42ea-a022-2dfe9ed56ad8
spec:
  acme:
    email: xxx
    http01: {}
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
status:
  acme:
    uri: https://acme-v02.api.letsencrypt.org/acme/acct/76984529
  conditions:
  - lastTransitionTime: "2020-05-29T13:31:11Z"
    message: The ACME account was registered with the ACME server
    reason: ACMEAccountRegistered
    status: "True"
    type: Ready


>>kubectl cert-manager inspect secret tls-secret-p9v2
...
Debugging:
        Trusted by this computer:       no: x509: certificate has expired or is not yet valid: current time 2021-08-24T07:03:32Z is after 2021-08-22T06:40:20Z
        CRL Status:     No CRL endpoints set
        OCSP Status:    Cannot check OCSP: error reading OCSP response: ocsp: error from server: unauthorized



 kubectl  describe secret tls-secret-p9v2
Name:         tls-secret-p9v2
Namespace:    supplier
Labels:       certmanager.k8s.io/certificate-name=tls-secret-p9v2
Annotations:  certmanager.k8s.io/alt-names: shipit-dev-p9v2.westeurope.cloudapp.azure.com
              certmanager.k8s.io/common-name: shipit-dev-p9v2.westeurope.cloudapp.azure.com
              certmanager.k8s.io/ip-sans:
              certmanager.k8s.io/issuer-kind: ClusterIssuer
              certmanager.k8s.io/issuer-name: letsencrypt-prod-p9v2

Type:  kubernetes.io/tls

Data
====
tls.key:  1679 bytes
ca.crt:   0 bytes
tls.crt:  5672 bytes


kubectl get order
NAME                         STATE   AGE
tls-secret-p9v2-4123722043   valid   24d

[(⎈ |shipit-k8s-dev:supplier)]$ k describe order tls-secret-p9v2-4123722043
Name:         tls-secret-p9v2-4123722043
Namespace:    supplier
Labels:       acme.cert-manager.io/certificate-name=tls-secret-p9v2
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Order
Metadata:
  Creation Timestamp:  2021-07-31T04:12:42Z
  Generation:          4
  Managed Fields:
    API Version:  certmanager.k8s.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .:
          f:acme.cert-manager.io/certificate-name:
        f:ownerReferences:
          .:
          k:{"uid":"a1dec741-0fe7-42be-99d2-176c3d4cdf38"}:
            .:
            f:apiVersion:
            f:blockOwnerDeletion:
            f:controller:
            f:kind:
            f:name:
            f:uid:
      f:spec:
        .:
        f:config:
        f:csr:
        f:dnsNames:
        f:issuerRef:
          .:
          f:kind:
          f:name:
      f:status:
        .:
        f:certificate:
        f:challenges:
        f:finalizeURL:
        f:state:
        f:url:
    Manager:    jetstack-cert-manager
    Operation:  Update
    Time:       2021-07-31T04:13:09Z
  Owner References:
    API Version:           certmanager.k8s.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  tls-secret-p9v2
    UID:                   a1dec741-0fe7-42be-99d2-176c3d4cdf38
  Resource Version:        143545958
  UID:                     a646985b-6d44-4c99-bb39-ceb6c4919047
Spec:
  Config:
    Domains:
      shipit-dev-p9v2.westeurope.cloudapp.azure.com
    http01:
      Ingress Class:  nginx
  Csr:                MIIC3zCCAccCAQAwTzEVMBMGA1UEChMMY2VydC1tYW5hZ2VyMTYwNAYDVQQDEy1zaGlwaXQtZGV2LXA5djIud2VzdGV1cm9wZS5jbG91ZGFwcC5henVyZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCdqF08foRx7qVCU4YpVLHxodqMJp1h10l0s89MUVK7C2IWwHdQ5w2BjUB12gT6T6NK9ZhJEzzYtLk18wFAojKUOFjuwF5Kklh+Qe6rFiZNNJ2+uDN/WhCLylbsXjHzQ+N3XMZ0jhGv+72XQyeK/X8jurMmVk5dSZbYP0ysk7w7gSFjjpeN2EIpYcnp2rCjTU+ksfeJ04DDm84hN9snMpGKspIhTFBphCQQgScPO9Fx+S5NVG/ScoM0CLSYiQVB0oPYUaw84O/lNC7kq/UWERli2pNy9Gnxdw2g37nFTj2uvPGGbPE1WBTFtdzkFWMepaw1l25X1//Nsap3zuZY0C3jAgMBAAGgSzBJBgkqhkiG9w0BCQ4xPDA6MDgGA1UdEQQxMC+CLXNoaXBpdC1kZXYtcDl2Mi53ZXN0ZXVyb3BlLmNsb3VkYXBwLmF6dXJlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAWEqfGuYcgf2ujby+K9MK+9q/r0cajo4q0JM6ZkBQQGb88b/nwxa7sr4n7hnlpKhXdLPp5QoeBMr3UM7Nwc7PrYxOws9v51mq3ekUOARgO4/4eJw4agFf8KKQLjtkFr2Q3OFJp6GuYKDCo2+Z1jqs76v7ReKlBoVhMtxOkjykQJheFQzg7ezGshE5trXh3NL/FaaThp1vP+qp8nDnq1YXkvOyaoc7u4X2sl831FTPcv3tsQJJzrOPlZPUJcgCC9cZiCTwqdttaJFRobTEGSk+pzc54C6eRQv9muto8D29Eg2G9f9xDSJULT6WbZWL6gzbJ/5pu3ep+V+cB43f5H+Sqg==
  Dns Names:
    shipit-dev-p9v2.westeurope.cloudapp.azure.com
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-prod-p9v2
Status:
  Certificate:  LS0tLS1CRUdJTiBDRVJUSUZJ.....
  Challenges:
    Authz URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/17660284180
    Config:
      http01:
        Ingress Class:  nginx
    Dns Name:           shipit-dev-p9v2.westeurope.cloudapp.azure.com
    Issuer Ref:
      Kind:      ClusterIssuer
      Name:      letsencrypt-prod-p9v2
    Key:         AxP1pv5I087QVyKXIkGyT5pqlD4Aa-UYmJHAOgzHPu4.mIcOL5pBlkZJSpSUslpjJTC_hFunxNRCEA82VcfFAHE
    Token:       AxP1pv5I087QVyKXIkGyT5pqlD4Aa-UYmJHAOgzHPu4
    Type:        http-01
    URL:         https://acme-v02.api.letsencrypt.org/acme/chall-v3/17660284180/Sh057Q
    Wildcard:    false
  Finalize URL:  https://acme-v02.api.letsencrypt.org/acme/finalize/75003870/13444902230
  State:         valid
  URL:           https://acme-v02.api.letsencrypt.org/acme/order/75003870/13444902230
Events:          <none>