This relates to DST Root CA X3 Expiration (September 2021)
When searching online for a fix to apply on an older server (Debian 8 in my case) that does call to sites encrypted with letsencrypt with curl
, they now seem to fail with the following message:
Example:
curl -fsSL https://deb.nodesource.com/setup_14.x | bash -
Fails silently, then trying it manually and removing the silent flag and bash pipe like this:
curl -L https://deb.nodesource.com/setup_14.x
curl: (60) SSL certificate problem: certificate has expired
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Trying the following commands doesn’t solve the issue:
apt update
apt install -y ca-certificates openssl
update-ca-certificates
What can I do!? (answering my own question) ⬇️
2
Answers
disclaimer; I'm no security expert (I know things, but you do you). Make sure you understand what you do before applying whatever fix shared here
Fix by upgrading your instances
Upgrade your instances. This problem won't happen on debian 9 or higher.
In the following example, I had this problem on a
ruby:2.4.1
docker image which is based on Debian 8 (could be considered old). Upgrading to more recent docker image fixes this issue. Uprading to a more recent Debian version should also fix the issue.I confirmed it does not happen when using
ruby:2.7.0
docker image based on Debian 11 as shown here:Fix for Debian 8 by commenting DST_Root_CA_X3.crt from /etc/ca-certificates.conf
Even if ISRG Root X1 is in place, if DST Root CA X3 is still present and in use, its verification seems to happen first so we can get rid of it by doing this:
ca-certificates
package/mozilla/DST_Root_CA_X3.crt
from/etc/ca-certificates.conf
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
is there (it should be)update-ca-certificates
Example directly on your instance
Example
Dockerfile
:Fix using dpkg-reconfigure ca-certificates
As stated in the comments, you can also fix this interactively using the following command on the instance (requires
ca-certificates
package installed):Then disable
mozilla/DST_Root_CA_X3.crt
from the list.Conclusion
If you'd like to learn more, you should read Scott Helme's post: Let's Encrypt's Root Certificate is expiring!
You can now
curl
letsencrypt sites safely with these safety glasses: 🥽It doesn’t get clear how "old" your Debian is.
Having the latest Linux OS will not cause this issue.
Consider before everything upgrading your distro to the latest version!
The following worked on my older Debian and Ubuntu systems.
To clarify the issue is with CA Root X3 certificate part of the "cacert-bundle".
As of today the "cacert-bundle" can be found here: https://curl.se/docs/caextract.html
as part of the bundle https://curl.se/ca/cacert.pem.
The expired certificate is:
Which is used to verify peer in curl calls to websites using Let’s Encrypt issued certificates.
Find which CA Certificates bundle is being used by CURL:
Result:
Near the end you should see the certificate bundle. In this case: /etc/ssl/certs/ca-certificates.crt
An option to fix this issue is to replace that CA certificate bundle, but that may not be a global solution for your OS and other affected apps and packages (E.g.: php libcurl).
Now to dissect the Root CA bundle certificate, use:
NOTE: The cert location (/etc/ssl/certs/ca-certificates.crt) in the command is from the "strace" step above.
This will return long list with expiration dates for all Root CA certificates in the bundle.
In this case, our expired certificate is "Not After : Sep 30 14:01:15 2021 GMT".
You can remove the | grep "Not After" to see the full decrypted list.
This the step-by-step method to troubleshoot the cacert bundle.
If this happens in the future you know what to do.
Now let’s get to fixing the issue.
FIX
Let’s Encrypt originally used the “DST Root CA X3” CA Root certificate.
Let’s encrypt now uses “ISRG Root X1” and “ISRG Root X2” as Root CA’s and “Let’s Encrypt R3” as an intermediate certificate.
To fix this issue, you need to add the 2 new Root CAs to your server or device:
Intermediate Certificate (PEM format):
Install Root CA Certificate on Linux:
NODEJS
Node.js 7.3.0 (and the LTS versions 6.10.0 and 4.8.0) added NODE_EXTRA_CA_CERTS environment variable for you to pass the CA certificate file.
Alternatively you can blacklist/remove the DST certificate from the CA cert bundle for your OS.
DEBIAN/UBUNTU
You can use the dpkg to reconfigure in Ubuntu/Debian.
Next step retest your curl call from the terminal.
You should no longer see the CURL error:
Another option (Ubuntu/Debian):
The list of CAs is stored in the file /etc/ca-certificates.conf. You can edit this file manually and run:
To automate it:
RHEL/CENTOS
In RedHat add that cert to the ca-trust blacklist: /etc/pki/ca-trust/source/blacklist
Add the file to your blacklist location: /etc/pki/ca-trust/source/blacklist
Run
sudo update-ca-trust
Source: https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/
Let’s Encrypt formal article: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/