Debian Question posted in
The oficial documentation can be found here.

Getting permission denied even as root inside the docker container – Debian

Connecting to running docker container as a root still gets Operation not permitted error when trying to apt-get update, yet I can still see sensitive file like /etc/passwd. Below are my configurations and also the error message from apt-get update. My host operating system is Ubuntu 18.04.3. My docker version is Docker version 19.03.5, build 633a0ea838

I create a container with the following Dockerfile

FROM python:3.8-slim-buster
RUN useradd -ms /bin/bash andrej
WORKDIR /home/andrej
COPY . /home/andrej/

RUN apt-get update && 
    apt-get install -y gcc && 
    pip install -r requirements.txt && 
    apt-get remove -y gcc && apt-get -y autoremove

RUN chown andrej:andrej pycurl && 
    chmod 0744 pycurl

USER andrej
ENTRYPOINT ["uwsgi"]
CMD ["--ini", "uwsgi.ini"]

starting it with docker compose looking like this:

version: "3.3"

services:

  andrej-cv:
    build: ./andrej_cv
    container_name: andrej-cv
    restart: always
    security_opt:
      - no-new-privileges
    expose:
      - 5000
    healthcheck:
      test: ./pycurl --host=127.0.0.1 --port=5050 --uri=/health_check
      interval: 1m30s
      timeout: 10s
      retries: 3

My docker daemon config:

{
    "icc": false,
    "userns-remap": "default",
    "log-driver": "syslog",
    "live-restore": true,
    "userland-proxy": false,
    "no-new-privileges": true
}

I connect to the container with following command (as root):
docker exec -it -u root <container_hash> /bin/bash but when I try to update I got the following:

root@ed984abff684:/home/andrej# apt-get update 
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
Hit:1 http://deb.debian.org/debian buster InRelease
Ign:2 http://deb.debian.org/debian buster-updates InRelease
Err:4 http://deb.debian.org/debian buster-updates Release
  Could not open file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_Release - open (13: Permission denied) [IP: 151.101.36.204 80]
Hit:3 http://security-cdn.debian.org/debian-security buster/updates InRelease
rm: cannot remove '/var/cache/apt/archives/partial/*.deb': Permission denied
Reading package lists... Done
W: chown to _apt:root of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chmod 0700 of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chown to _apt:root of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chmod 0700 of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
W: Download is performed unsandboxed as root as file '/var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_Release - PrepareFiles (13: Permission denied)
E: The repository 'http://deb.debian.org/debian buster-updates Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: Problem unlinking the file /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_buster_updates_InRelease - PrepareFiles (13: Permission denied)

In the container /etc/subuid and /etc/subgid look like this (both):

andrej:100000:65536

On the host /etc/subuid and /etc/subgid look like this (both):

andrej:100000:65536
dockremap:165536:65536

Apparmor is running on Ubuntu host with following status (only docker-default profile):

andrej@machine:/etc/apparmor.d$ sudo aa-status 
apparmor module is loaded.
38 profiles are loaded.
36 profiles are in enforce mode.
   /sbin/dhclient
   /snap/core/8268/usr/lib/snapd/snap-confine
   /snap/core/8268/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/evince
   /usr/bin/evince-previewer
   /usr/bin/evince-previewer//sanitized_helper
   /usr/bin/evince-thumbnailer
   /usr/bin/evince//sanitized_helper
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/cups/backend/cups-pdf
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/cups-browsed
   /usr/sbin/cupsd
   /usr/sbin/cupsd//third_party
   /usr/sbin/ippusbxd
   /usr/sbin/tcpdump
   docker-default
   libreoffice-senddoc
   libreoffice-soffice//gpg
   libreoffice-xpdfimport
   man_filter
   man_groff
   snap-update-ns.core
   snap-update-ns.gnome-calculator
   snap-update-ns.gnome-characters
   snap-update-ns.gnome-logs
   snap-update-ns.gnome-system-monitor
   snap.core.hook.configure
   snap.gnome-calculator.gnome-calculator
   snap.gnome-characters.gnome-characters
   snap.gnome-logs.gnome-logs
   snap.gnome-system-monitor.gnome-system-monitor
2 profiles are in complain mode.
   libreoffice-oopslash
   libreoffice-soffice
17 processes have profiles defined.
14 processes are in enforce mode.
   docker-default (1101) 
   docker-default (1102) 
   docker-default (1111) 
   docker-default (1600) 
   docker-default (1728) 
   docker-default (1729) 
   docker-default (1730) 
   docker-default (1731) 
   docker-default (1732) 
   docker-default (1798) 
   docker-default (1799) 
   docker-default (1800) 
   docker-default (1801) 
   docker-default (1802) 
0 processes are in complain mode.
3 processes are unconfined but have a profile defined.
   /sbin/dhclient (491) 
   /usr/sbin/cups-browsed (431) 
   /usr/sbin/cupsd (402)

Selinux seems to be disabled as there is not /etc/selinux/config file and getenfoce and sestatus command are not available.

Also su andrej command run as root (where andrej is unprivileged user in the container) errors out with su: cannot set groups: Operation not permitted

Tags:

6

Answers


  1. 0

    I had exatly the same issues when running
    Ubuntu-16.04-based container in rootless Podman with Manjaro as the host system.

    TL;DR: try to rebuild the image. That helped in my case.

    The issue is likely that Docker cannot map the /var/lib/apt/lists/package directory’s owner (_apt) UID to host’s UID namespace. This might happen if the /etc/sub{u,g}id is modified after the image is pulled/built.

    This is only a guess but the reason might be that Docker performs the UID map first for the image and then modifies /etc/sub{u,g}id resulting in different UID map rules -> Docker cannot map the user inside the container.

    You can verify this by running docker inspect <image name> and checking the directories in "LowerDir" part. In one of those there should exist a directory var/lib/apt/lists/package with UID outside of the range specified for dockremap in /etc/sub{u,g}id. The exact command for podman was podman image inspect <image name> --format '{{.GraphDriver.Data.LowerDir}}' but the CLI APIs of Podman and Docker should be identical so the same command should work with Docker also.

    E.g. I had an entry tlammi:100000:65536 in /etc/sub{u,g}id but /var/lib/apt/lists/package was owned by UID 165538 in host side which is outside of range [100000, 100000 + 65536).

    Login or Signup to reply.
  2. 0

    We run into the same issue with rootless podman.

    We changed the subuid/subgid range of the user. This means one would need to fix the files stored with the old ranges or just delete the temporary files from the container storage directory:

    $ podman info|grep GraphRoot
GraphRoot: /opt/tmp/container_graphroot/.local/share/containers/storage
$ rm -rf /opt/tmp/container_graphroot/.local/share/containers/storage/*
    Login or Signup to reply.
  3. 0

    This is not an answer yet (as of 2021-05-10) but my current research on this issue so far. Hopefully, it would give others hints about where to look further. Maybe I’ll come back to edit this post to make it a real answer.

    As far as I see, the "issue" is caused by the use of the security option no-new-privileges. Note that it is specified in OP’s docker-compose file and the Docker daemon’s configuration file.

    Here is its description in the Docker’s doc:

    –security-opt="no-new-privileges:true" Disable container processes from gaining new privileges

    …

    If you want to prevent your container processes from gaining additional privileges, you can execute the following command:

    $ docker run --security-opt no-new-privileges -it centos bash

    no-new-privileges is a flag that was added to Linux kernel since 3.5. Here is its doc:

    The no_new_privs bit (since Linux 3.5) is a new, generic mechanism to make it safe for a process to modify its execution environment in a manner that persists across execve. Any task can set no_new_privs. Once the bit is set, it is inherited across fork, clone, and execve and cannot be unset. With no_new_privs set, execve() promises not to grant the privilege to do anything that could not have been done without the execve call. For example, the setuid and setgid bits will no longer change the uid or gid; file capabilities will not add to the permitted set, and LSMs will not relax constraints after execve.

    Notice the last sentence of "the setuid and setgid bits will no longer change the uid or gid". This may be why you would see the following error messages:

    E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)

    I found an article that talks about it with examples clearly: Running Docker application containers more securely.

    My current thoughts:

    • I don’t call the failed "apt-get update" an "issue" or a "problem" because that should be an intentional behavior for security consideration. In other words, it’s a good thing.
    • Because the quoted kernel doc says "Once the bit is set, it … cannot be unset", I believe you won’t be able to "fix" it in the existing containers.
    • I don’t think removing no-new-privileges is the right solution. At least it’s not right before you fully discuss it with your team.
    • Alternatively, create a container without no-new-privileges for testing purpose only.
    Login or Signup to reply.
  4. 0

    NOTE: Only if you have docker and docker-compose installed
    If initially you had not been running as root and rebuilt the image, run a prune

    docker system prune -f
docker-compose up

    This then makes sure you’re running on a fresh build

    Login or Signup to reply.
  5. 0

    Not sure about Docker, but in kubernetes in runc container for me helps:

    1. Get root access to container
      List all containers
    minikube ssh docker container ls

    Connect to your container (use your container id from previous command instead of 44a7ad70d45b):

    minikube ssh "docker container exec -it -u 0 44a7ad70d45b /bin/bash"
    1. As root inside container:
    root@mycontainer:/# apt-config dump | grep Sandbox::User
APT::Sandbox::User "_apt";

root@mycontainer:/# cat <<EOF > /etc/apt/apt.conf.d/sandbox-disable
APT::Sandbox::User "root";
EOF

    Be sure, that result is valid:

    root@mycontainer:/# apt-config dump | grep Sandbox::User
APT::Sandbox::User "root";
    1. apt update and see
    Get:1 http://archive.ubuntu.com/ubuntu hirsute InRelease [269 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt hirsute-pgdg InRelease [16.7 kB]
...
    Login or Signup to reply.
  6. 0

    The docker container you describe is working as designed. With the default capabilities.

    Fast test if you have a missing capability.

    Run your container with Full/All container capabilities (--privileged)

    More explanations about capabilities from Docker documentation

    The Linux kernel is able to break down the privileges of the root user
    into distinct units referred to as capabilities. For example, the
    CAP_CHOWN capability is what allows the root user to make arbitrary
    changes to file UIDs and GIDs. The CAP_DAC_OVERRIDE capability allows
    the root user to bypass kernel permission checks on file read, write
    and execute operations. Almost all of the special powers associated
    with the Linux root user are broken down into individual capabilities.

    This breaking down of root privileges into granular capabilities
    allows you to:

    Remove individual capabilities from the root user account, making it less powerful/dangerous.
Add privileges to non-root users at a very granular level.

    Capabilities apply to both files and threads. File capabilities allow
    users to execute programs with higher privileges. This is similar to
    the way the setuid bit works. Thread capabilities keep track of the
    current state of capabilities in running programs.

    The Linux kernel lets you set capability bounding sets that impose
    limits on the capabilities that a file/thread can gain.

    Docker imposes certain limitations that make working with capabilities
    much simpler. For example, file capabilities are stored within a
    file’s extended attributes, and extended attributes are stripped out
    when Docker images are built. This means you will not normally have to
    concern yourself too much with file capabilities in containers.

    It is of course possible to get file capabilities into containers at runtime, however this is not recommended.

    In an environment without file based capabilities, it’s not possible
    for applications to escalate their privileges beyond the bounding set
    (a set beyond which capabilities cannot grow). Docker sets the
    bounding set before starting a container. You can use Docker commands
    to add or remove capabilities to or from the bounding set.

    By default, Docker drops all capabilities except those needed, using a
    whitelist approach.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search