Establish SSH connection from Jenkins container to SSH server container, I can establish with password login but can't establish with private key – CentOS

The Story:

I am following a Udemy course for learning Jenkins with docker. The lab is that I setup a SSH server container & a Jenkins container, put them in the same docker network. Then, I go into the jenkins container to establish ssh connection to the SSH server container.

The issue:

I can establish SSH connection from jenkins container to the SSH server container with password login, however, it fails to establish the SSH -i with the key pair I generated.

The overview of my directory and files:

Under my working space of my local machine, I have:

• a docker-compose.yml (details see below)
• a directory named centos/,
• Inside centos/ I have a Dockerfile for building the SSH server image.

The steps I tried and their results:

Step 1, I firstly created the docker-compose.yml for the two containers

version: '3'
services:
  jenkins:
    container_name: jenkins
    image: jenkins/jenkins
    ports:
      - "8080:8080"
    volumes:
      - $PWD/jenkins_home:/var/jenkins_home
    networks:
      - net
  remote_host:
    container_name: remote_host
    image: remote-host
    build: 
      context: centos7
    networks:
      - net
networks:
  net:

As you can see above, the remote_host service refers to the SSH server image which is built from a Dockerfile under centos/ directory.

Step 2, so here is my Dockerfile under centos/ :

FROM centos

RUN yum -y install openssh-server

RUN useradd remote_user && 
    echo remote_user:1234 | chpasswd && 
    mkdir /home/remote_user/.ssh && 
    chmod 700 /home/remote_user/.ssh

COPY remote-key.pub /home/remote_user/.ssh/authorized_keys

RUN chown remote_user:remote_user -R /home/remote_user/.ssh/ && 
    chmod 600 /home/remote_user/.ssh/authorized_keys

RUN ssh-keygen -A
RUN rm -rf /run/nologin
CMD /usr/sbin/sshd -D

Step 3. As you can see in above Dockerfile, I COPY a public key remote-key.pub from my local host to the container and named it authorized_keys. I generated the key pair under the centos/ folder of my local host by command:

ssh-keygen -t rsa -m PEM -f remote-key

Step 4. After all above are done. I run command docker-compose build to build images.

Step 5. Finally I run docker-compose up to bring up containers.

Both jenkins & remote_host(the ssh server) containers are up and running successfully.

Step 6. Then I go inside the jenkins container by docker exec -it jenkins bash, in the bash terminal of the container, I do ssh connection:

jenkins@7551f2fa441d:/$ ssh remote_user@remote_host
remote_user@remote_host's password: 

After input the passowrd 1234 (Defined in the Dockerfile), I successfully established the SSH connection from jenkins container to the SSH server container. But I would like to establish the connection via key pairs instead of password login.

Step 7, since the SSH server container has the public key copied, so, I copied the private key from my local host to the jenkins container by command:

docker cp remote-key jenkins:/tmp/

So, now the jenkins container’s /tmp/ directory contains the private key.

Step 8, Now I try to use the private key to establish the SSH connection to the ssh server container. I go inside the jenkins by docker exec -it jenkins bash, then, I run command:

jenkins@7551f2fa441d:/$ ssh -i /tmp/remote-key remote_user@remote_host
Load key "/tmp/remote-key": Permission denied
remote_user@remote_host's password: 

As you can see above, inside jenkins container, the bash prompt tells me the key "/tmp/remote-key": Permission denied. Right after that it asks me to input password, then I input password 1234, and the SSH connection is established.

So, why I can’t use the private key to establish the SSH connection from the jenkins container to the ssh server container but only login with password works?

==== Update =====

Inspired by the comment from @agentsmith under his answer, I checked the ownership of the /tmp/remote-key inside my jenkins container:

drwxr-xr-x 2 jenkins   jenkins       4096 Sep 20 13:01 hsperfdata_jenkins
drwxr-xr-x 2 root      root          4096 Feb  2  2020 hsperfdata_root
drwxr-xr-x 2 jenkins   jenkins       4096 Sep 20 13:01 jetty-0_0_0_0-8080-war-_-any-190970179478026794.dir
drwxr-xr-x 2 jenkins   jenkins       4096 Sep 18 20:55 jetty-0_0_0_0-8080-war-_-any-878046537266404011.dir
-rw------- 1 245867976 1349604816    1679 Sep 18 20:53 remote-key
-rw-r--r-- 1 jenkins   jenkins    3167976 Sep 18 20:55 winstone3001500689590881345.jar
-rw-r--r-- 1 jenkins   jenkins    3167976 Sep 20 13:01 winstone8218655308653013358.jar

As you can see above, all other files have user and group ownership of jenkins (one file with root) except the remote-key file. So I tried to change the ownership by:

jenkins@7551f2fa441d:/$ chown jenkins:jenkins /tmp/remote-key 
chown: changing ownership of '/tmp/remote-key': Operation not permitted 

However the chown command tells me operation not permitted. 🙁 What should I do now?