I have read many links similar to my issue, but none of them were helping me to resolve the issue.
Similar Links:
- Failed to exec into the container due to permission issue after executing ‘systemctl daemon-reload’
- OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
- CI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
- OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
- Fail to execute docker exec
- OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "open /proc/self/fd: no such file or directory": unknown
Problem Description:
I have created a new Kubernetes cluster using Kubespray
. When I wanted to execute some commands in one of containers I faced to the following error:
Executed Command
kubectl exec -it -n rook-ceph rook-ceph-tools-68d847b88d-7kw2v -- sh
Error:
OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/1: operation not permitted: unknown
command terminated with exit code 126
I have also logged in to the node, which runs the pod, and try executing the container using docker exec
command, but the error was not changed.
Workarounds:
-
As I have found, the error code (126) implies that the permissions are insufficient, but I haven’t faced this kind of error (like executing
sh
) in Docker or Kubernetes. -
I have also checked whether
SELinux
is enabled or not (as it has been said in the 3rd link).apt install policycoreutils sestatus # Output SELinux status: disabled
-
In the 5th link, it was said to check whether you have updated the kernel, and I didn’t upgrade anything on the nodes.
id; stat /dev/pts/0 # output uid=0(root) gid=0(root) groups=0(root) File: /dev/pts/0 Size: 0 Blocks: 0 IO Block: 1024 character special file Device: 18h/24d Inode: 3 Links: 1 Device type: 88,0 Access: (0600/crw-------) Uid: ( 0/ root) Gid: ( 5/ tty) Access: 2022-08-21 12:01:25.409456443 +0000 Modify: 2022-08-21 12:01:25.409456443 +0000 Change: 2022-08-21 11:54:47.474457646 +0000 Birth: -
-
Also tried
/bin/sh
instead ofsh
or/bin/bash
, but not worked and the same error occurred.
Can anyone help me to find the root cause of this problem and then solve it?
2
Answers
I have found the root cause after about 3 months. Kubespray has been used to install the cluster. Kubespray has a strange bug when installing OCI (in my case Docker). The configuration for Docker is not applied, and the changes are not applied properly. @xirehat suggested restarting Docker as a solution to the Kubespray problem.
This issue may relate to docker, first drain your node.
Second, SSH to the node and restart docker service.
At the end try to execute your command.