Now that Ubuntu 22.04 is released I did a clean install on one of our jenkins-workers to test it but I can’t get the docker ssh-agent to work properly. It can no longer identify that it’s running inside a container, so whenever a job is launching that uses docker I can see in the console "Jenkins-worker-X does not seem to be running inside a container", followed by the pipeline failing.
I know from before that jenkins uses cgroup information to detect whether it’s running in a container, so e.g. executing cat /proc/self/cgroup
in a container should result in a list of lines ending with /docker/<container-id>
, which is then used by Jenkins to detect the container. However, once I installed Ubuntu 22.04 the cgroup information no longer contains the /docker/<container-id>
which causes the jenkins agent to think it’s running on bare metal.
Even executing the official image has the same problem, i.e. docker run jenkins/ssh-agent:jdk11
followed by docker exec <container-id> cat /proc/self/cgroup
ends up with a list without the container hashes on my machine.
How do I troubleshoot this? Has something changed from Ubuntu 21.10 to 22.04 that causes this problem? Is some extra configuration necessary?
I’m running latest Ubuntu 22.04 (5.15.0-27-generic), Docker version 20.10.12, build 20.10.12-0ubuntu4.
Any help would be appreciated!
EDIT: I now realized that the same thing happens in 21.10 if you upgrade all packages to the latest version (and use the latest jenkins/ssh-agent image), so the cause might be in one of the upgraded packages
2
Answers
It turned out that the problem was related to cgroup v2 after all. It seems that when using v2 the cgroup namespace is private by default when you create a container, in my case the Jenkins agents, which caused the container id to not be available in
/proc/self/cgroup
.The easy solution is to run the docker container with
--cgroupns host
as suggested in another question here. When I did that Jenkins could once again detect the container it's running inside.An update was probably released for Ubuntu 21.10 switching to cgroup v2, just as I posted the question, since I could later reproduce the issue there as well.
If the Jenkins container is being run with Docker Compose, you can supply the
cgroup
parameter mentioned in the other answer in the compose file: https://docs.docker.com/compose/compose-file/05-services/#cgroupAlternatively, if you have control over the Docker daemon running Jenkins, you can set the
default-cgroupns-mode
flag in your Docker Daemon config tohost
. Note that this will apply to all containers on the host, though.