Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Ubuntu – can't push to local docker registry via ssh tunnel

I’ve rented a server on which I’d like to host my own Docker container registry. The server uses Ubuntu 24.04, docker and docker compose are installed.

Now, I’ve created this docker-compose.yml to run a registry on the server:

version: '3'
services:
  registry:
    image: registry:2
    ports:
      - "127.0.0.1:5000:5000"
    environment:
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
    volumes:
      - ./data:/var/lib/registry

I’m running this with docker compose up.
Then I’ve set up an ssh tunnel from my client to the server ssh -L 5000:localhost:5000 user@server.

Now, as a sanity check, I’m running this command on my client:

$ curl -X GET http://127.0.0.1:5000/v2/_catalog
{"repositories":[]}

The output looks reasonable, an empty list of repositories. On the server I’m running docker compose up without -d flag, to look at the logging output and it says:

registry-1  | time="2024-10-08T10:24:16.455126761Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="127.0.0.1:5000" http.request.id=133d3573-9be6-48fe-9102-ffdac2706c93 http.request.method=GET http.request.remoteaddr="172.18.0.1:35336" http.request.uri="/v2/_catalog" http.request.useragent="curl/8.4.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.285188ms http.response.status=200 http.response.written=20 
registry-1  | 172.18.0.1 - - [08/Oct/2024:10:24:16 +0000] "GET /v2/_catalog HTTP/1.1" 200 20 "" "curl/8.4.0"

Now I’d like to tag and push an image. I’m running the following commands on the client:

$ docker tag hello-world 127.0.0.1:5000/test:latest
$ docker push 127.0.0.1:5000/test:latest
The push refers to repository [127.0.0.1:5000/test]
Get "http://127.0.0.1:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused

I’ve also tried tagging this with localhost instead of 127.0.0.1 but the result is the same.

To sanity check whether the registry is working, I’ve tried the same on the server:

root@Ubuntu-2404-noble-amd64-base ~ # docker tag hello-world 127.0.0.1:5000/test:latest
root@Ubuntu-2404-noble-amd64-base ~ # docker push 127.0.0.1:5000/test:latest
The push refers to repository [127.0.0.1:5000/test]
ac28800ec8bb: Pushed latest: digest: sha256:d37ada95d47ad12224c205a938129df7a3e52345828b4fa27b03a98825d1e2e7 size: 524

So this works, the registry seems to work correctly. As another sanity check, let’s repeat the curl command on the client:

$ curl -X GET http://127.0.0.1:5000/v2/_catalog
{"repositories":["test"]}

So this seems to correctly point to the docker registry on the remote server. The ssh tunnel seems to work.
But the exact same commands that can tag and push an image on the server, fail with a connection error on the client.

I’m using MacOS on the client. I’ve set insecure registries in the settings of the Docker engine:

  "insecure-registries": [
    "127.0.0.1:5000",
    "localhost:5000"
  ]

At this point I’m at a loss. Can you advise me how to proceed with troubleshooting this?

Tags:

2

Answers


  1. 0

    The following is a proof of concept. Use https , not SSH.

    Architecture

    +---------+      https     +---------+
| clientA |  ------------> | serverB |
+---------+                +---------+
macOS-Client               Ubuntu-2404-noble-amd64-base
                           yourServer Public IP
                           yourServerFQDN name

    Please confirm the following information about your Server:

    • yourServer Public IP
    • yourServerFQDN name
    +---------+                              +---------+
| clientA |  ------------------------->  | serverB |
+---------+                              +---------+
                                              |
                                              | (docker-compose)
                       +----------------------+-----------------+
                       |port 443 (https)                        | port 5000
               +------------+                              +-----------------+
               |  Nginx     |   <--------------------->    | Docker Registry |
               +------------+                              +-----------------+
                       |  (Front-end)                           |  (Back-end)
                       +----------------------------------------+

    The following will use the machine in my internal environment as an example, so the IP and Server FQDN name are examples. Please adjust them according to your own Sercer settings.

    For my example:

    • yourServer Public IP (192.168.56.112)
    • yourServerFQDN name (demoserver.demo.local)

    Since I don’t have DNS, to simplify things, I directly modified the file contents of /etc/hosts.

    /etc/hosts

    ADD this line to /etc/hosts file

    192.168.56.112  demoserver.demo.local

    (Both clientA and serverB need to be modified.)

    serverB

    create a directory my_docker_registry as your project directory.

    Project Tree

    my_docker_registry <directory>
├── docker-compose.yml
├── my-demo-registry <directory>
│   ├── auth <directory>
│   │   └── htpasswd
│   └── data <directory>
└── nginx
    ├── certs <directory>
    ├── Dockerfile
    └── nginx.conf

    config registry

    All instructions are executed in the my_docker_registry (project root directory).

    mkdir -p my-demo-registry/auth
mkdir -p my-demo-registry/data

    add user id and password

    sudo apt install -y apache2-utils

# c create file
htpasswd -Bc my-demo-registry/auth/htpasswd demo01user

# second user do not create file
htpasswd -B my-demo-registry/auth/htpasswd demo02user

    create two users:

    • demo01user
    • demo02user

    config nginx

    All instructions are executed in the my_docker_registry (project root directory).

    mkdir -p nginx/certs

    create ssl certificate

    The following contents are adjusted according to the contents of https://goharbor.io/docs/2.5.0/install-config/configure-https/ to generate ssl credentials.

    The following will switch the instruction execution directory to the nginx/certs directory.

    cd nginx/certs

    Step 01 : Generate a CA certificate private key.

    openssl genrsa -out ca.key 4096

    Step 02 : Generate the CA certificate.

    Note, please adjust the content of the following commands according to the FQDN name of the server you set (for example: demoserver.demo.local ).

    openssl req -x509 -new -nodes -sha512 -days 3650 
  -subj "/C=CN/ST=NewYork/L=NewYork/O=example/OU=Personal/CN=demoserver.demo.local" 
  -key ca.key 
  -out ca.crt

    Step 03 : Generate a Server Certificate

    Step 03-1 : Generate a private key.
    openssl genrsa -out demoserver.demo.local.key 4096
    Step 03-2 : Generate a certificate signing request (CSR).
    openssl req -sha512 -new 
  -subj "/C=CN/ST=NewYork/L=NewYork/O=example/OU=Personal/CN=demoserver.demo.local" 
  -key demoserver.demo.local.key 
  -out demoserver.demo.local.csr
    Step 03-3 : Generate an x509 v3 extension file.
    cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=demoserver.demo.local
DNS.2=demoserver
DNS.3=localhost
EOF
    Step 03-4 : Use the v3.ext file to generate a certificate for your Server host.
    openssl x509 -req -sha512 -days 3650 
  -extfile v3.ext 
  -CA ca.crt 
  -CAkey ca.key 
  -CAcreateserial 
  -in demoserver.demo.local.csr 
  -out demoserver.demo.local.crt

    Step 04 : Convert demoserver.demo.local.crt to demoserver.demo.local.cert, for use by Docker.

    openssl x509 
  -inform PEM 
  -in demoserver.demo.local.crt 
  -out demoserver.demo.local.cert

    Step 05 : Copy the server certificate, key and CA files into the Docker certificates folder on your server host. You must create the appropriate folders first.

    sudo mkdir -p /etc/docker/certs.d/demoserver.demo.local

sudo cp demoserver.demo.local.cert /etc/docker/certs.d/demoserver.demo.local/

sudo cp demoserver.demo.local.key /etc/docker/certs.d/demoserver.demo.local/

sudo cp ca.crt /etc/docker/certs.d/demoserver.demo.local/

sudo systemctl restart docker

    nginx/Dockerfile

    FROM nginx:alpine

COPY nginx.conf /etc/nginx/nginx.conf

RUN mkdir -p /etc/nginx/certs
COPY certs/demoserver.demo.local.crt /etc/nginx/certs/
COPY certs/demoserver.demo.local.key /etc/nginx/certs/

    nginx/nginx.conf

    events {
    worker_connections 1024;
}

http {
    server {
        listen 443 ssl;
        
        # Replace with your domain name
        server_name demoserver.demo.local;

        # SSL Certificate Path
        ssl_certificate /etc/nginx/certs/demoserver.demo.local.crt;
        
        # SSL key path
        ssl_certificate_key /etc/nginx/certs/demoserver.demo.local.key;

        location / {
            # Reverse proxy to Docker Registry
            # proxy_pass http://localhost:5000;
            proxy_pass http://registry:5000;
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

    docker-compose.yml

    version: '3.3'

services:
  registry:
    image: registry:2
    ports:
        # For internal access only
      - "5000:5000"  
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd      
    volumes:
      - ./my-demo-registry/auth:/auth
      - ./my-demo-registry/data:/var/lib/registry
# Data Volume      
#     - registry_data:/var/lib/registry  

  nginx:
    build: ./nginx
    ports:
      # Map to port 443 on the host
      - "443:443"
    depends_on:
      - registry

# volumes:
#   registry_data:

    Start

    docker-compose build

docker-compose up

    check

    docker ps

    You will see two containers being executed. One is nginx and the registry.

    Test

    login

    docker login https://demoserver.demo.local

    input username and password. (my example: demo01user)

    will return Login Succeeded

    • pull hello-world
    docker pull hello-world

docker tag hello-world demoserver.demo.local/demo01user/hello-world:0.0.1

docker push demoserver.demo.local/demo01user/hello-world:0.0.1
    • list images
    docker image ls

    get

    REPOSITORY                                     TAG       IMAGE ID       CREATED         SIZE
...
demoserver.demo.local/demo01user/hello-world   0.0.1     d2c94e258dcb   17 months ago   13.3kB
hello-world                                    latest    d2c94e258dcb   17 months ago   13.3kB

    clientA – your macOS

    • Now copy the nginx/certs directory on serverB to your clientA host (your macOS)

    • Execute the following instructions

    Note that related instructions may need to be adjusted under macOS.

    sudo mkdir -p /etc/docker/certs.d/demoserver.demo.local
sudo cp demoserver.demo.local.cert /etc/docker/certs.d/demoserver.demo.local/
sudo cp demoserver.demo.local.key /etc/docker/certs.d/demoserver.demo.local/
sudo cp ca.crt /etc/docker/certs.d/demoserver.demo.local/
    
sudo systemctl restart docker

    Note: This is executed on clientA which is your macOS.

    Since I don’t have DNS, to simplify things, I directly modified the file contents of /etc/hosts.

    /etc/hosts

    ADD this is /etc/hosts

    192.168.56.112  demoserver.demo.local

    Now, on clientA execute:

    docker run demoserver.demo.local/demo01user/hello-world:0.0.1

    You should be able to see the execution results of hello-world` retrieved from serveB.

    You can also check the image on your local machine.

    docker image ls
    Login or Signup to reply.
  2. 0

    I’m using MacOS on the client.

    localhost points to multiple things with Docker on MacOS. Inside the container it points to that container. On your MacOS host, it points to your host. But to the docker engine, it points to the VM where docker is running.

    From the docker engine, to point to the IP of your MacOS host, you would point to host.docker.internal instead of localhost. Or as another alternative, you could run the ssh proxy inside of a container. This is also why running the registry inside of a container works.

    For more details, see: From inside of a Docker container, how do I connect to the localhost of the machine?

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search