Debian Question posted in
The oficial documentation can be found here.

Docker daemon fails to process logs – Debian

Docker daemon is unable to parse the json log and throws an unexpected EOF error. We are investigating the root cause of the issue.

Environment:

  • Debian 4.9.189-3+deb9u2
  • Docker engine 19.03.5

Docker daemon log:

Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070677515+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=345
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070695689+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=346
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070712630+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=347
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070732299+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=348
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070755016+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=349
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070773699+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=350

Docker daemon configuration

"log-driver": "json-file",
"log-opts": {
  "mode": "non-blocking",
  "max-size": "500m",
  "max-file": "3"
}

Storage space is available on partition where docker writes its logs.

Your insight in the matter is most welcome.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    The aforementioned issue will cause an abnormal spike in CPU usage of the docker (version <= 19) process and constantly stay at that level until you fix the logs.

    To fix the logs & cpu usage, you've to truncate all the container logs (since we are unable to determine which container has faulty logs). It works only if your logs are in json format:

    truncate -s 0 /var/lib/docker/containers/*/*-json.log

    Please see Andre's answer for a permanent fix.


  2. 0

    Issue=Docker Engine Error json-file: fix sporadic unexpected EOF errors

    I had spent some time researching and trying to find errors in the docker
    logs.

    There is a bug in docker engine 20.10.5 which is the version running in docker. A Github PR fixes this. https://github.com/moby/moby/pull/42104

    I found the error while running

    $ journalctl -u docker.service
level=warning msg="got error while decoding json" error="unexpected EOF" retries=19999

    Docker driver for logs has a bug and it wont handle some log formatting. The bug then floods the system’s log with retries messages. This situation generates a very annoying CPU load.

    Docker engine latest is 20.10.7. The issue has been fixed in release Docker engine 20.10.6 https://docs.docker.com/engine/release-notes/#20106

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search