Question 47023 in Debian Question posted in
The oficial documentation can be found here.

How to connect fluentd to opendistro for elasticsearch – Debian

I have created a docker file :

FROM fluentd:v1.14.0-debian-1.0
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.1.2"]
USER fluent

From which I create a container in my compose file:

version: '3'
services:
  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    links:
      - "elasticsearch"
    restart: always
    container_name: fluentd
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    networks:
      - network
  elasticsearch:
    image: docker pull amazon/opendistro-for-elasticsearch:latest
    container_name: elasticsearch
    environment:
      - cluster.name=elasticsearch
      - node.name=elasticsearch
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - network
  kibana:
    image: docker pull amazon/opendistro-for-elasticsearch-kibana:latest
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://elasticsearch:9200
      ELASTICSEARCH_HOSTS: https://elasticsearch:9200
    networks:
      - network 
volumes:
  elasticsearch:

networks:
  network:
    driver: bridge
    name: network

And this is my fluent.conf

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match *.**>
  @type copy

  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    user admin
    password admin
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    scheme https
    ssl_verify false
  </store>

  <store>
    @type stdout
  </store>
</match>

For some reason the elasticsearch and the kibana start without issues, yet fluentd cannot connect. This is the error I am getting

 *

2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.2/lib/fluent/plugin/elasticsearch_index_template.rb:52:in `retry_operate'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.2/lib/fluent/plugin/out_elasticsearch.rb:487:in `handle_last_seen_es_major_version'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.2/lib/fluent/plugin/out_elasticsearch.rb:339:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:187:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:110:in `block in configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `each'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/out_copy.rb:39:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:187:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:132:in `add_match'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:74:in `block in configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `each'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/root_agent.rb:149:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/engine.rb:105:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/engine.rb:80:in `run_configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/supervisor.rb:716:in `block in run_worker'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/supervisor.rb:968:in `main_process'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/supervisor.rb:708:in `run_worker'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/bin/fluentd:15:in `<top (required)>'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/bin/fluentd:23:in `load'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/bin/fluentd:23:in `<main>'
    fluentd          | 2021-11-04 16:47:35 +0000 [info]: Worker 0 finished unexpectedly with status 1
    fluentd          | 2021-11-04 16:47:43 +0000 [info]: adding match pattern="*.**" type="copy"
    fluentd          | 2021-11-04 16:47:45 +0000 [error]: #0 unexpected error error_class=Elasticsearch::UnsupportedProductError error="The client noticed that the server is not a supported distribution of Elastic

I am not sure if the plugin used for fluentd to connect to elasticsearch is eligible to also connect to OpenDistro for elasticsearch. If any other information is needed please ask.
Any help would be much appreciated.

Tags:

4

Answers


  1. Chosen as BEST ANSWER
    0

    the problem is that the plugin works with lower version of elasticsearch, after refactoring my docker file everything worked as a charm:

    FROM fluentd:v1.14.0-debian-1.0
USER root
RUN gem install elasticsearch -v 7.13.3 
&& gem install fluent-plugin-elasticsearch --no-document
USER fluent

  2. 0

    Use stable latest versions of plugins elasticsearch and fluent-plugin-elasticsearch.
    My worked Dockerfile for example:

    FROM fluent/fluentd:v1.12.0-debian-1.0
    USER root
    RUN ["gem", "install", "elasticsearch", "–no-document", "–version", "7.13.3"]
    RUN ["gem", "install", "fluent-plugin-elasticsearch", "–no-document", "–version", "5.0.5"]
    USER fluent

    Login or Signup to reply.
  3. 0

    I was testing the connection with an elasticsearch 7.10.2 server

    I removed the fluentd plugins with recentd version to 8.x and install the plugins for the version I’m using (7)
    This section help me to debug the issue

    RUN fluent-gem uninstall --force fluent-plugin-elasticsearch
RUN fluent-gem uninstall --force elasticsearch
RUN fluent-gem uninstall --force elastic_ruby_console
RUN fluent-gem uninstall -a --force elasticsearch-api
RUN fluent-gem uninstall -a --force elastic-transport
RUN fluent-gem uninstall -a --force elasticsearch-xpack
RUN fluent-gem install elasticsearch -v 7.10
RUN fluent-gem install fluent-plugin-elasticsearch
    Login or Signup to reply.
  4. 0

    it is work

    version: '3.7'
services:
  fluentd:
    image: cr.fluentbit.io/fluent/fluent-bit
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search