Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Ubuntu – Kubernetes on aws ec2 – private registry certificate signed by unknown authority

Issue Summary

I have a Kubernetes cluster 1.30 running on 3 AWS EC2 servers, 1 master node, and 2 worker nodes. I am trying to deploy a private container registry using the registry:2.8.2 image of the CNCF distribution project. I have followed the steps listed below, but I am getting the error "x509 certificate signed by unknown authority" while I am trying to perform a docker login to the registry.

Note: I am using self-signed certificates.

Steps to Reproduce

  1. Create self-signed certificates and user authentication. The TLS certificates use OpenSSL:

    mkdir registry
cd registry
mkdir certs
mkdir auth

openssl req -x509 -newkey rsa:4096 -days 365 -nodes -sha256 -keyout certs/tls.key -out certs/tls.crt -subj "/CN=my-registry" -addext "subjectAltName = DNS:my-registry"

  2. Use htpasswd to add user authentication for registry access. The credentials for the private registry are myuser/mypasswd:

    docker run --entrypoint htpasswd httpd:2 -Bbn myuser mypasswd > auth/htpasswd

  3. Create a secret to mount the certificates (secret type tls):

    kubectl create secret tls certs-secret --cert=/home/ubuntu/day42/registry/certs/tls.crt --key=/home/ubuntu/day42/registry/certs/tls.key

  4. Create a secret for authentication:

    kubectl create secret generic auth-secret --from-file=/home/ubuntu/day42/registry/auth/htpasswd

  5. Create the local directory and apply PV PVC.

  6. Create the registry pod.

apiVersion: v1
kind: Pod
metadata:
  name: docker-registry-pod
  labels:
    app: registry
spec:
  containers:
  - name: registry
    image: registry:2.8.2
    volumeMounts:
    - name: repo-vol
      mountPath: "/var/lib/registry"
    - name: certs-vol
      mountPath: "/certs"
      readOnly: true
    - name: auth-vol
      mountPath: "/auth"
      readOnly: true
    env:
    - name: REGISTRY_AUTH
      value: "htpasswd"
    - name: REGISTRY_AUTH_HTPASSWD_REALM
      value: "Registry Realm"
    - name: REGISTRY_AUTH_HTPASSWD_PATH
      value: "/auth/htpasswd"
    - name: REGISTRY_HTTP_TLS_CERTIFICATE
      value: "/certs/tls.crt"
    - name: REGISTRY_HTTP_TLS_KEY
      value: "/certs/tls.key"
  volumes:
  - name: repo-vol
    persistentVolumeClaim:
      claimName: registry-pvc
  - name: certs-vol
    secret:
      secretName: certs-secret
  - name: auth-vol
    secret:
      secretName: auth-secret

  1. Create the service pod.

  2. Export the environment variables:

    export REGISTRY_NAME="my-registry"
Master-->export REGISTRY_IP="10.100.175.12"
# (Above IP is the NodePort service IP)

  3. Add the below entry to /etc/hosts of all nodes:

    10.100.175.12 my-registry

  4. Copy tls.crt to /usr/local/share/ca-certificates on all the nodes and run the following command:

    sudo update-ca-certificates

  5. Try to login to the registry:

    docker login my-registry:5000 -u myuser -p mypasswd

Error Details

Error response from daemon: Get "https://my-registry:5000/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

Output of crictl info

crictl info |grep -i tls
    "enableTLSStreaming": false,
      "tlsCertFile": "",
      "tlsKeyFile": ""
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Phew! After spending 3 days with this issue, I finally figured it out.

    Below are the steps I have taken to fix the issue:

    Ensure All Nodes Trust the Self-Signed Certificate:

    • Verify that tls.crt is copied to /usr/local/share/ca-certificates on all nodes
    • Run sudo update-ca-certificates on each node.

    Configure Docker to Trust the Self-Signed Certificate:

    Docker needs to be explicitly configured to trust your self-signed certificate. You must copy the tls.crt to Docker's trusted certificates directory.

        sudo mkdir -p /etc/docker/certs.d/my-registry:5000
    sudo cp /path/to/your/tls.crt /etc/docker/certs.d/my- 
    registry:5000/ca.crt

    Note: Replace /path/to/your/tls.crt with the path to your tls.crt file.

    Run docker login again

    docker login https://my-registry:5000 -u myuser -p mypasswd
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

  2. 0

    For the certificates like the SSL ones, you need to add the entire certificate chain into the certificate file you are using with nginx to run the registry. Many operating systems don’t trust the intermediate CAs, just the root CA, so you need to fill in the missing steps between the certificate for your host and the root CA that is trusted by the OS.

    In the email you received your certificate with, you should also find links to the intermediate CAs and the root CA. Open the docker-registry.crt file, scroll to the bottom, and append the intermediate CAs and, finally, the root CA certificate for the PositiveSSL chain. Once you’ve done that, restart nginx. You should now be good to go.

    Note: If you own a domain name you can use Let’s Encrypt to obtain a trusted certificate at zero cost.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search