There’s a very strange issue, the docker daemon can’t response the building request and other commands like docker info
. But if I build image from an exist image Dockerfile FROM some/exist/image
, it works well, docker version
works well too. We have about 56 VMs to build images, the building command is just like this::
export DOCKER_HOST = '<my_host>'
docker build -t <tag> - < context.tar.gz
The client output below, it seems that it gets stuck on extrating:
[test]$ docker build -t test .
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM busybox
latest: Pulling from library/busybox
7c9d20b9b6cd: Extracting 32.77kB/760.8kB
^C
But the cpu and memory are no problem, also there’s still a lot of disk space.
[~]# top
top - 10:18:09 up 48 days, 20:20, 1 user, load average: 1.00, 1.03, 1.05
Tasks: 167 total, 1 running, 166 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.7 us, 0.3 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8006376 total, 603984 free, 612016 used, 6790376 buff/cache
KiB Swap: 4192928 total, 4186560 free, 6368 used. 6749588 avail Mem
Log:
I used systemctl to start dockerd and it’s strange that I can’t see recent log about dockerd, it seems that dockerd gets stuck from 16:04:08:
cat /var/log/messages|grep docker
2019-10-28 16:04:07 dockerd[1549]: time="2019-10-28T16:04:07+08:00" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/d584a3957292ac1d3edfea2ec57abcf2dbe79465f9bcd41183b0ccd9dac3bb01/shim.sock" debug=false pid=18014
2019-10-28 16:04:08 dockerd[1549]: time="2019-10-28T16:04:08+08:00" level=info msg="shim reaped" id=d584a3957292ac1d3edfea2ec57abcf2dbe79465f9bcd41183b0ccd9dac3bb01
2019-10-28 16:04:08 dockerd[1549]: time="2019-10-28T16:04:08.081553011+08:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2019-10-28 19:15:00 dockerd[1549]: time="2019-10-28T19:15:00.826993765+08:00" level=error msg="Not continuing with pull after error: context canceled"
2019-10-28 19:17:49 dockerd[1549]: time="2019-10-28T19:17:49.216471516+08:00" level=error msg="Handler for POST /v1.38/build returned error: Error processing tar file(exit status 1): unexpected EOF"
2019-10-28 19:42:18 dockerd[1549]: time="2019-10-28T19:42:18.954682094+08:00" level=error msg="Not continuing with pull after error: context canceled"
2019-10-28 19:51:51 dockerd[1549]: time="2019-10-28T19:51:51.611278576+08:00" level=info msg="Attempting next endpoint for pull after error: manifest unknown: manifest unknown"
2019-10-28 19:52:03 dockerd[1549]: time="2019-10-28T19:52:03.076089163+08:00" level=info msg="Attempting next endpoint for pull after error: manifest unknown: manifest unknown"
2019-10-28 19:55:41 dockerd[1549]: time="2019-10-28T19:55:41.605964950+08:00" level=info msg="Attempting next endpoint for pull after error: Get https://xx/centos/manifests/6: no basic auth credentials"
2019-10-28 19:55:44 dockerd[1549]: time="2019-10-28T19:55:44.298856115+08:00" level=info msg="Attempting next endpoint for pull after error: Get https://xx/centos/manifests/6: no basic auth credentials"
Docker processes:
[~]# ps -ef|grep docker
root 1549 1 6 Oct09 ? 1-05:45:10 /usr/bin/dockerd
root 1557 1549 0 Oct09 ? 00:42:20 docker-containerd --config /var/run/docker/containerd/containerd.toml
root 2574 1549 0 Oct09 ? 00:00:01 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8090 -container-ip x.x.x.x -container-port 8080
root 2581 1557 0 Oct09 ? 00:00:41 docker-containerd-shim -namespace moby -workdir /opt/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/56e9432a55304bc61d284d3a9da15272d292c1493d33d4499e481a0d64ff53e4 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
root 15630 15049 0 10:13 pts/0 00:00:00 grep --color=auto docker
Docker version:
[~]$ docker version
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:23:03 2018
OS/Arch: linux/amd64
Experimental: false
Proc Stack
[root@vm ~]# cat /proc/1549/stack
[<ffffffff810f8564>] futex_wait_queue_me+0xc4/0x120
[<ffffffff810f90d9>] futex_wait+0x179/0x280
[<ffffffff810fb1de>] do_futex+0xfe/0x5b0
[<ffffffff810fb710>] SyS_futex+0x80/0x180
[<ffffffff8169d53d>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
I’ve seen the github issue like https://github.com/docker/for-win/issues/813 and https://github.com/moby/moby/issues/12823 , but I can’t get a correct way to solve this.
How can I solve this problem? Any help would be greatly appreciated.
System
Centos
Kernel: 3.10.0-514.41.1.el7 x86_64 GNU/Linux
2
Answers
Its hard to reproduce your problem. But here are some ideas, how you can solve the problem.
Sometimes a restart via
systemctl restart dockerd
solves problems.I saw a problem of corrupted installation of docker. It leads to strange behaviors they never get 1:1 reproduced. Just reinstall docker on the maschine.
Many other factors can produce this problem. Maybe there are problems with your centos. In one of your links, somebody discovered that a antivirus-system was the bad one (see https://github.com/docker/for-win/issues/813#issuecomment-431031402).
you should start docker daemon with debug mode, like
FYI, configure and debug docker daemon