Question 388606 in Nginx Question posted in
The official documentation can be found here.

Nginx – Docker swarm and fluentd logging driver

I tried to record some logs for front-end nginx-based containers using fluentd docker logging driver, but failed.

I ended up with following configuration for fluentd (located in /tmp/fluentd/fluent.conf):

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match **-**>
  @type stdout
</match>

and following docker swarm manifest:

version: '3.8'

networks:
  base:

services:
  fluentd:
    image: fluent/fluentd:v1.17.0-1.0
    volumes:
      - /tmp/fluentd:/fluentd/etc
    ports:
      - 24224:24224
    networks:
      - base
  test-curl:
    image: alpine
    command: sh -c "apk add curl && while true; do curl fluentd:24224; sleep 5; done"
    depends_on:
      - fluentd
    networks:
      - base
  test-service:
    image: nginx:1.25
    depends_on:
      - fluentd
    logging:
      driver: fluentd
      options:
        # fluentd-address: host.docker.internal:24224 # this line routes queries via docker host
        fluentd-address: fluentd:24224 # this line tries to send logs directly to fluentd container
        tag: something
    networks:
      - base

This configuration runs fluentd service and two test services. First (with alpine-based curl test) works fine. Second, with fluentd driver, does not. It fails to start with following message:

> docker service ps fluentd-test_test-service --no-trunc
...
<...>    _ fluentd-test_test-service.1   nginx:1.25@sha256:<...> docker-desktop   Shutdown        Failed 22 seconds ago   "starting container failed: failed to create task for container: failed to initialize logging driver: dial tcp: lookup fluentd: i/o timeout"
...

Looks like for some reason it can’t see fluentd container in the very same network.
But, as I am commenting out marked line and uncomment line with host.docker.internal address, everything works as intended.

Running under WSL2 (2.1.5.0, kernel 5.15.146.1-2), Docker 26.1.1, Windows 10.0.19045.4412

What I tried:

  • upgrading docker
  • upgrading wsl
  • upgrading fluent/fluentd
  • diagnosing fluentd (open ports, accessibility, dns resolution — works fine)
  • inspecting all docker objects (looks absolutely normal)
  • asking chatgpt

Please, any ideas will be appreciated.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Looks like desired behavior is not achievable with fluentd driver, as it is written in docs:

    fluentd Writes log messages to fluentd (forward input). The fluentd daemon must be running on the host machine.

    https://docs.docker.com/config/containers/logging/configure/#supported-logging-drivers


  2. 0

    Dockers log driver is implemented by docker itself, and as such has no access to containers or container networking. So when you specify the address for docker to talk to, you need to specify a url the docker daemon itself can reach. As such, if you have fluentd running in a container, or directly as a daemon on the host, you would specify localhost:24224 as the target for logs.

    docker network names will not work, and if host.docker.internal works it must be because there is an entry in /etc/hosts aliasing it to localhost.

    In the case of docker swarm, you only need 1 fluentd service as "localhost:24224" will be routed to the fluentd container from any swarm node.

    There are some caveats that mean you should never run fluentd as a container – docker expects logging drivers to be present, and if it cannot log will literally pause critical operations (containers that need logging will get stuck in starting states and other worse things) until the log container is fixed.

    Use docker log drivers when you have an off-swarm highly available log target. Otherwise its best to stick to Dockers json file based logs, and tools like promtail/logstash etc. to to scrape them.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search