skip to Main Content

There is a container in my Kubernetes cluster which I want to debug.

But there is nonetstat, no ip and no apk.

Is there a way to upgrade this image, so that the common tools are installed?

In this case it is the nginx container image in a K8s 1.23 cluster.

2

Answers


  1. The whole point of using containers is to optimize the resource utilization in your cluster. The images used should only include the packages that are needed to run your app.

    The unwanted packages should be removed from your images (especially in prod) to reduce the compute utilization and to reduce the attack vector.

    This appears to be a stripped down image that has only the libraries needed to run that application.

    In order to debug, you will have to create a new container in the same pid and network namespace as the container you are trying to debug

    Build container first

    Dockerfile

    FROM alpine
    RUN apk update && apk add strace
    CMD ["strace", "-p", "1"]
    

    Build

    $ docker build -t strace .
    

    Run

    docker run -t --pid=container:<targetContainer> 
      --net=container:targetContainer 
      --cap-add sys_admin 
      --cap-add sys_ptrace 
      strace
    strace: Process 1 attached
    futex(0xd72e90, FUTEX_WAIT, 0, NULL
    

    https://rothgar.medium.com/how-to-debug-a-running-docker-container-from-a-separate-container-983f11740dc6

    Login or Signup to reply.
  2. Alpine is a stripped-down version of the image to reduce the footprint. So the absence of those tools is expected. Although since Kubernetes 1.23, you can use the kubectl debug command to attach a debug pod to the subject pod.
    Syntax:

    kubectl debug -it <POD_TO_DEBUG> --image=ubuntu --target=<CONTAINER_TO_DEBUG> --share-processes
    

    Example:
    In the below example, the ubuntu container is attached to the Nginx-alpine pod, requiring debugging. Also, note that the ps -eaf output shows nginx process running and the cat /etc/os-release shows ubuntu running. The indicating process is shared/visible between the two containers.

    ps@kube-master:~$ kubectl debug -it nginx --image=ubuntu --target=nginx --share-processes
    Targeting container "nginx". If you don't see processes from this container, the container runtime doesn't support this feature.
    Defaulting debug container name to debugger-2pgtt.
    If you don't see a command prompt, try pressing enter.
    root@nginx:/# ps -eaf
    UID          PID    PPID  C STIME TTY          TIME CMD
    root           1       0  0 19:50 ?        00:00:00 nginx: master process nginx -g daemon off;
    101           33       1  0 19:50 ?        00:00:00 nginx: worker process
    101           34       1  0 19:50 ?        00:00:00 nginx: worker process
    101           35       1  0 19:50 ?        00:00:00 nginx: worker process
    101           36       1  0 19:50 ?        00:00:00 nginx: worker process
    root         248       0  1 20:00 pts/0    00:00:00 bash
    root         258     248  0 20:00 pts/0    00:00:00 ps -eaf
    root@nginx:/# 
    

    Debugging as ubuntu as seen here, this arm us with all sort of tools:

    root@nginx:/# cat /etc/os-release
    NAME="Ubuntu"
    VERSION="20.04.3 LTS (Focal Fossa)"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 20.04.3 LTS"
    VERSION_ID="20.04"
    HOME_URL="https://www.ubuntu.com/"
    SUPPORT_URL="https://help.ubuntu.com/"
    BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
    PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
    VERSION_CODENAME=focal
    UBUNTU_CODENAME=focal
    root@nginx:/# 
    

    In case ephemeral containers need to be enabled in your cluster, then you can enable it via feature gates as described here.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search