skip to Main Content

I have read many links similar to my issue, but none of them were helping me to resolve the issue.

Similar Links:

  1. Failed to exec into the container due to permission issue after executing ‘systemctl daemon-reload’
  2. OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
  3. CI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
  4. OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
  5. Fail to execute docker exec
  6. OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "open /proc/self/fd: no such file or directory": unknown

Problem Description:

I have created a new Kubernetes cluster using Kubespray. When I wanted to execute some commands in one of containers I faced to the following error:

Executed Command
kubectl exec -it -n rook-ceph rook-ceph-tools-68d847b88d-7kw2v -- sh
Error:

OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/1: operation not permitted: unknown
command terminated with exit code 126

I have also logged in to the node, which runs the pod, and try executing the container using docker exec command, but the error was not changed.

Workarounds:

  • As I have found, the error code (126) implies that the permissions are insufficient, but I haven’t faced this kind of error (like executing sh) in Docker or Kubernetes.

  • I have also checked whether SELinux is enabled or not (as it has been said in the 3rd link).

    apt install policycoreutils
    sestatus
    # Output
    SELinux status:                 disabled
    
  • In the 5th link, it was said to check whether you have updated the kernel, and I didn’t upgrade anything on the nodes.

    id; stat /dev/pts/0
    # output
    uid=0(root) gid=0(root) groups=0(root)
      File: /dev/pts/0
      Size: 0               Blocks: 0          IO Block: 1024   character special file
    Device: 18h/24d Inode: 3           Links: 1     Device type: 88,0
    Access: (0600/crw-------)  Uid: (    0/    root)   Gid: (    5/     tty)
    Access: 2022-08-21 12:01:25.409456443 +0000
    Modify: 2022-08-21 12:01:25.409456443 +0000
    Change: 2022-08-21 11:54:47.474457646 +0000
     Birth: -
    
  • Also tried /bin/sh instead of sh or /bin/bash, but not worked and the same error occurred.

Can anyone help me to find the root cause of this problem and then solve it?

2

Answers


  1. Chosen as BEST ANSWER

    I have found the root cause after about 3 months. Kubespray has been used to install the cluster. Kubespray has a strange bug when installing OCI (in my case Docker). The configuration for Docker is not applied, and the changes are not applied properly. @xirehat suggested restarting Docker as a solution to the Kubespray problem.


  2. This issue may relate to docker, first drain your node.

    kubectl drain <node-name>
    

    Second, SSH to the node and restart docker service.

    systemctl restart docker.service
    

    At the end try to execute your command.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search