Question 221553 in Docker Question posted in
The official documentation can be found here.

Docker – Container fails to start because "/var/run/secrets/kubernetes.io/serviceaccount" read-only file system: unknown

I have an EKS cluster. When I deploy my container it fails to start with the following message:

Error: failed to create containerd task: failed to create shim task: OCI runtime
create failed: runc create failed: unable to start container process: error during
container init: error mounting
"/var/lib/kubelet/pods/039da982-b1d6-4e91-88d0-d8a5b9791c17/volumes/kubernetes.io~projected/kube-api-access-4vwhm"
to rootfs at "/var/run/secrets/kubernetes.io/serviceaccount":
mkdir /run/containerd/io.containerd.runtime.v2.task/k8s.io/app/rootfs/run/secrets/kubernetes.io:
read-only file system: unknown

Description of the pod:

Name:             app-6b6bf4dbfb-dbhwf
Namespace:        default
Priority:         0
Service Account:  default
Node:             ip-10-0-0-20.eu-central-1.compute.internal/10.0.0.20
Start Time:       Fri, 28 Jul 2023 13:00:50 +0200
Labels:           app=app
                  pod-template-hash=6b6bf4dbfb
Annotations:      <none>
Status:           Running
IP:               10.0.0.25
IPs:
  IP:           10.0.0.25
Controlled By:  ReplicaSet/app-6b6bf4dbfb
Containers:
  app:
    Container ID:  containerd://21810e232495a61bf0381cfdf75ca668c351e17635f12ea0e3340d683d49a7f2
    Image:         xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/app:43c51e0bba7d4d16322c53fc64805fb8ae1c1d03
    Image ID:      xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/app@sha256:cd25ee9556334356e1a4bb9985408a9f900d48e2e7984db05dcbc94b90134e32
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /app
      --es_db_url=http://10.0.2.52:9200
      --registry_address=consul:8500
      --registry=consul
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/039da982-b1d6-4e91-88d0-d8a5b9791c17/volumes/kubernetes.io~projected/kube-api-access-4vwhm" to rootfs at "/var/run/secrets/kubernetes.io/serviceaccount": mkdir /run/containerd/io.containerd.runtime.v2.task/k8s.io/21810e232495a61bf0381cfdf75ca668c351e17635f12ea0e3340d683d49a7f2/rootfs/run/secrets/kubernetes.io: read-only file system: unknown
      Exit Code:    128
      Started:      Thu, 01 Jan 1970 01:00:00 +0100
      Finished:     Fri, 28 Jul 2023 13:27:03 +0200
    Ready:          False
    Restart Count:  10
    Limits:
      cpu:     1
      memory:  1524Mi
    Requests:
      cpu:     100m
      memory:  1524Mi
    Environment:
      DAILY_INVITATION_LIMIT:          10
    Mounts:
      /run/secrets from service-acc-secret (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4vwhm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  service-acc-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  service-acc-secret
    Optional:    false
  kube-api-access-4vwhm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  27m                    default-scheduler  Successfully assigned default/app-6b6bf4dbfb-dbhwf to ip-10-0-0-20.eu-central-1.compute.internal
  Normal   Pulled     27m                    kubelet            Successfully pulled image "xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaw
s.com/app:43c51e0bba7d4d16322c53fc64805fb8ae1c1d03" in 1.263487759s (1.263518276s including waiting)
  Normal   Pulled     27m                    kubelet            Successfully pulled image "xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/app:43c51e0bba7d4d16322c53fc64805fb8ae1c1d03" in 94.189474ms (94.21938ms including waiting)
  Normal   Pulled     27m                    kubelet            Successfully pulled image "xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/app:43c51e0bba7d4d16322c53fc64805fb8ae1c1d03" in 111.835016ms (111.857053ms including waiting)
  Normal   Created    27m (x4 over 27m)      kubelet            Created container app
  Warning  Failed     27m (x4 over 27m)      kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/039da982-b1d6-4e91-88d0-d8a5b9791c17/volumes/kubernetes.io~projected/kube-api-access-4vwhm" to rootfs at "/var/run/secrets/kubernetes.io/serviceaccount": mkdir /run/containerd/io.containerd.runtime.v2.task/k8s.io/app/rootfs/run/secrets/kubernetes.io: read-only file system: unknown
  Normal   Pulled     27m                    kubelet            Successfully pulled image "xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/app:43c51e0bba7d4d16322c53fc64805fb8ae1c1d03" in 109.735988ms (109.760967ms including waiting)
  Normal   Pulling    26m (x5 over 27m)      kubelet            Pulling image "xxxxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/app:43c51e0bba7d4d16322c53fc64805fb8ae1c1d03"
  Warning  BackOff    2m55s (x118 over 27m)  kubelet            Back-off restarting failed container

Container’s base image:

FROM alpine:3.18

RUN apk --update upgrade && 
    apk add curl ca-certificates && 
    update-ca-certificates && 
    apk add --no-cache tzdata 
    rm -rf /var/cache/apk/*

ADD main /app
ADD client_secret.json /client_secret.json

It started to happen when I upgraded the version of the alpine image from an old alpine:3.2. As I experimented it breaks at version alpine:3.7. I checked what is changed in version 3.7, there are a bunch of security updates. I guess one of them might be related. But nothing helpful.
My secrets are mounted to a different directory /run/secrets. I don’t know why Kubernetes cannot create the /var/run/secrets/kubernetes.io/serviceaccount directory after alpine:3.7 version.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I found the solution. The problem is I used the /run/secrets directory to mount my secret. Before alpine:3.6 this works but after alpine:3.7 there is a symlink to the /run directory. In alpine:3.7:

    $ ls -al /var/run
lrwxrwxrwx    1 root     root             4 Jun 14 15:03 /var/run -> /run

    So the /run and /var/run directory is the same. But the /var/run/secrets directory is already used by Kubernetes to mount the service accounts to access the Kube API and it is read-only.

    So the solution is to use something different than /run or /var/run directory to mount.


  2. 0

    read-only file system

    Not sure, but maybe it is a permissions related issue?
    Try to chmod to write enable before you mount.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search