skip to Main Content

I have a k8s cluster with an ingress nginx as a reverse proxy. I am using letsencrypt to generate TLS certificate

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ******
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx

Everything worked fine for months. Today,

$ curl -v --verbose https://myurl

returns

* Rebuilt URL to: https://myurl/
*   Trying 51.103.58.**...
* TCP_NODELAY set
* Connected to myurl (51.103.58.**) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

For 2 other people on my team, error is the same and I have the same error when I use Postman (expired certificate).

But for another one, we get no error :

*   Trying 51.103.58.**...
* TCP_NODELAY set
* Connected to myurl (51.103.58.**) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=myurl
*  start date: Jul 24 07:15:13 2021 GMT
*  expire date: Oct 22 07:15:11 2021 GMT
*  subjectAltName: host "myurl" matched cert's "myurl"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd9be00d600)
> GET / HTTP/2
> Host: myurl
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx/1.19.1
< date: Thu, 30 Sep 2021 16:11:23 GMT
< content-type: application/json; charset=utf-8
< content-length: 56
< vary: Origin, Accept-Encoding
< access-control-allow-credentials: true
< x-xss-protection: 1; mode=block
< x-frame-options: DENY
< strict-transport-security: max-age=15724800; includeSubDomains
< x-download-options: noopen
< x-content-type-options: nosniff
< etag: W/"38-3eQD3G7Y0vTkrLR+ExD2u5BSsMc"
<
* Connection #0 to host myurl left intact
{"started":"2021-09-30T13:30:30.912Z","uptime":9653.048}* Closing connection 0

When I use my web browser to go to the website, everything works fine and the certificate is presented as valid and for now, I get no error in prod or staging environment. (same error on staging)

Has anyone an explanation on this ?

3

Answers


  1. Warning! Please plan OS upgrade path. The below advice should be applied only in emergency situation to quickly fix a critical system.

    Your team missed OS update or ca-certificates package update.
    Below solution works on old Debian/Ubuntu systems.

    First check if you have offending DST Root CA X3 cert present:

    # grep X3 /etc/ca-certificates.conf 
    mozilla/DST_Root_CA_X3.crt
    

    Make sure the client OS have the proper ISRG Root X1 present too:

    # grep X1 /etc/ca-certificates.conf 
    mozilla/ISRG_Root_X1.crt
    

    This is going to disable X3:

    # sed -i '/^mozilla/DST_Root_CA_X3/s/^/!/' /etc/ca-certificates.conf && update-ca-certificates -f
    

    Try curl https://yourdomain now, should pass.

    Again, plan an upgrade please.

    Login or Signup to reply.
  2. Even if not K8S related, main explanation is contained in :
    Sudden OpenSSL Error messages: error:14090086 using file_get_contents . I complete it with K8S related here.

    I fixed the same issue by upgarding my certbot and reissuing certificate with “ –preferred-chain ‘ISRG Root X1’

    you can do the same with options in the yaml of the cert issuer :
    see here : https://cert-manager.io/docs/configuration/acme/#use-an-alternative-certificate-chain

    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: letsencrypt
    spec:
      acme:
        server: https://acme-v02.api.letsencrypt.org/directory
        preferredChain: "ISRG Root X1"
    
    Login or Signup to reply.
  3. This is related to the expired DST Root CA X3, which expired Sep 30 14:01:15 2021 GMT.

    The DST CA Root X3 certificate is part of the "cacert-bundle".
    As of today the "cacert-bundle" can be found here: https://curl.se/docs/caextract.html
    as part of the bundle https://curl.se/ca/cacert.pem.

    The expired certificate is:

    Certificate:
        Data:
        Version: 3 (0x2)
        Serial Number:
        44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
    Validity
        Not Before: Sep 30 21:12:19 2000 GMT
        Not After : Sep 30 14:01:15 2021 GMT
        Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
        Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
        Public-Key: (2048 bit)
    

    Which is used to verify peer in curl calls to websites using Let’s Encrypt issued certificates.

    Here’s a detailed solution to your problem: https://stackoverflow.com/a/69411107/1549092

    Let’s Encrypt formal address of the issue can be found here: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search