Passwordless SSH from GitLab CI to Remote Server – CentOS

Just recently I stumbled on an SSH issue that I cannot figure out what is missing. We use GitLab CI to build and deploy the project to one of our remote servers. As a part of the upgrade plan, we need to replace the degrading Debian 6 server with a new RHEL 7 server. I cannot get the passwordless SSH to work right from GitLab Runner to a remote machine.

I created a reproducible example in a Dockerfile, the IP of the remote server and the user is replaced with non-sensitive data.

FROM centos:7
RUN yum install -y epel-release
RUN yum update -y
RUN yum install -y openssh-clients
RUN useradd -m joe
RUN mkdir -p /home/joe/.ssh
COPY id_rsa_shared /home/joe/.ssh/id_rsa
RUN echo "Host *ntStrictHostKeyChecking non" >> /home/joe/.ssh/config
RUN ssh-keyscan 10.x.x.x >> /home/joe/.ssh/known_hosts
RUN chown -R joe:joe /home/joe/.ssh
USER joe
CMD ["/bin/bash"]

The file id_rsa_shared is created on local machine with the following command:

ssh-keygen -t rsa -b 2048 -f ./id_rsa_shared

ssh-copy-id -i ./id_rsa_shared [email protected]

This works on local. A simple ssh [email protected] uname -a in the docker container will output the following:

Linux newweb01p.company.local 3.10.0-1160.25.1.el7.x86_64 #1 SMP Tue Apr 13 18:55:45 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

However, if I commit this to a branch as GitLab CI script, as shown:

image: centos:7

stages:
  - deploy

dev-www:
  stage: deploy
  tags:
    - docker
  environment:
    name: dev-www
    url: http://dev-www.company.local
    variables:
      DEV_HOST: 10.x.x.x
      APP_ENV: dev
      DEV_USER: joe
  script:
    - whoami
    - yum install -y epel-release
    - yum update -y
    - yum install -y openssh-clients
    - useradd -m joe
    - mkdir -p /home/joe/.ssh
    - cp "./gitlab/known_hosts" /home/joe/.ssh/known_hosts
    - echo "$DEV_USER_OPENSSH_KEY" >> /home/joe/.ssh/id_rsa
    - echo "Host *ntStrictHostKeyChecking non" >> /home/joe/.ssh/config
    - chown -R joe:joe /home/joe/.ssh/
    - chmod 600 /home/joe/.ssh/*
    - chmod 700 /home/joe/.ssh
    - ls -Fsal /home/joe/.ssh
    - su - joe
    - ssh -oStrictHostKeyChecking=no "${DEV_USER}@${DEV_HOST}" uname -a
  when: manual

The pipeline will fail authentication as shown:

Running with gitlab-runner 13.12.0 (7a6612da)
    on docker.hqgitrunner01d.company.local K47w1s77
Preparing the "docker" executor
Using Docker executor with image centos:7 ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image centos:7 ...
Using docker image sha256:xxx for centos:7 with digest centos:7@sha256:xxxx ...
Preparing environment
Running on runner-k47w1s77-project-93-concurrent-0 via hqgitrunner01d.company.local...
Getting source from Git repository
Fetching changes...
Reinitialized existing Git repository in /builds/webversion3/API/.git/
Checking out 6a7c193b as tdr/psr4-composer...

Updating/initializing submodules recursively...
Executing "step_script" stage of the job script
Using docker image sha256:xxx for centos:7 with digest centos:7@sha256:xxx ...
$ whoami
root
$ useradd -m joe
$ mkdir -p /home/joe/.ssh
$ cp "./gitlab/known_hosts" /home/joe/.ssh/known_hosts
$ echo "$DEV_USER_OPENSSH_KEY" >> /home/joe/.ssh/id_rsa
$ echo "Host *ntStrictHostKeyChecking non" >> /home/joe/.ssh/config
$ chown -R joe:joe /home/joe/.ssh/*
$ chmod 600 /home/joe/.ssh/*
$ chmod 700 /home/joe/.ssh
$ ls -Fsal /home/joe/.ssh
total 16
0 drwx------ 2 root root  53  Apr  1 15:19 ./
0 drwx------ 3 joe  joe   74  Apr  1 15:19 ../
4 -rw------- 1 joe  joe   37  Apr  1 15:19 config
4 -rw------- 1 joe  joe 3414  Apr  1 15:19 id_rsa
8 -rw------- 1 joe  joe 6241  Apr  1 15:19 known_hosts
$ su - joe
$ ssh -oStrictHostKeyChecking=no "${DEV_USER}@${DEV_HOST}" uname -a
Warning: Permanently added '10.x.x.x' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

Cleaning up file based variables
ERROR: Job failed: exit code 1

Maybe there’s a step I missed because I get a ‘Permission denied, please try again’ message. How do I get Docker Executor to use passwordless SSH to a remote server?