skip to Main Content

We are using AWS Elasticsearch Service and fleuntd to push the logs of microservice. We have installed fluentd on EC2 instance using docker based configuration. We have followed the steps mentioned in the https://docs.fluentd.org/container-deployment/docker-compose and also it was working pretty fine till last week. There is no change in the configuration of both elastcicsearch service and ec2 instance we are provisioning. Below is the error logs we are getting.

2021-08-05 20:35:57 +0000 [error]: #0 unexpected error
error_class=NoMethodError error="undefined method
host_unreachable_exceptions' for #<Elasticsearch::Transport::Client:0x00007fe5231b5e08>" 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:41:in rescue in retry_operate’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:39:in
retry_operate' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:487:in handle_last_seen_es_major_version’ 2021-08-05 20:35:57 +0000
[error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:339:in
configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in configure’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:132:in
add_match' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in block in configure’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in
each' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in configure’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:146:in
configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in configure’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in
run_configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:714:in block in run_worker’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:966:in
main_process' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:706:in run_worker’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:364:in
<top (required)>' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in
require' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in <top
(required)>’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/bin/fluentd:23:in load' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/bin/fluentd:23:in

Below is the configuration used.

docker-compose.yml

version: '3'
services:
  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    ports:
      - "8880:8880"
      - "8880:8880/udp"

Dockerfile

FROM fluent/fluentd:v1.13.3
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.5"]
USER fluent

fluentd.conf

<source>
  @type forward
  port 8880
  bind 0.0.0.0
</source>

<match **>
    host ES_HOT
    port 443
    scheme https
    logstash_format true
    logstash_prefix "fluentd-logs"
    logstash_dateformat "%Y%m%d"
    user "elastic"
    password xxxxxx
    flush_interval 10s
    fail_on_detecting_es_version_retry_exceed false
    verify_es_version_at_startup false
    max_retry_get_es_version 2
    default_elasticsearch_version 7.10
    request_timeout 60s
    reload_on_failure true
</match>

Our configuration with AWS Elasticsearch Service 7.10, Fluentd v1.12.0-debian-1.0 and fluent-plugin-elasticsearch 4.3.3, was working perfectly fine till last week. One of the ec2 instance which was provisioned 10 days back, was able to connect and push the logs as well. But when we restarted that instance its also stopped working with the same error. We have Terraform scripts that provision all infrastructure. Later, we have tried different versions combinations as well. But nothing worked out.

Thanks in advance

2

Answers


  1. I had a similar issue. Digging into the matter, I found that the fluent-plugin-elasticsearch plugin doesn’t version-lock its dependencies, including elasticsearch-transport and elasticsearch-api, and that a recent update somewhere in one of those led to some kind of incompatibility.

    The fix that worked for me was to explicitly install the dependencies at the latest issue-free version to avoid having the fluent-plugin-elasticsearch install pull the latest issue-causing version.

    Proposed Modification to your Dockerfile

    FROM fluent/fluentd:v1.13.3
    USER root
    RUN ["gem", "install", "elasticsearch", "--no-document", "--version", "7.13.3"]
    RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.5"]
    USER fluent
    
    Login or Signup to reply.
  2. I launched an AWS Elasticsearch 7.10 managed service cluster and connected an EKS cluster to it with this Fluentd docker image:

    fluent/fluentd-kubernetes-daemonset:v1.12.0-debian-elasticsearch7-1.0

    Deploy manifest:

    piVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: fluentd
      namespace: kube-system
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      selector:
        matchLabels:
          k8s-app: fluentd-logging
          version: v1
      template:
        metadata:
          labels:
            k8s-app: fluentd-logging # This label will help group your daemonset pods
            version: v1
        spec:
          serviceAccountName: fluentd
          tolerations:
            - key: node-role.kubernetes.io/master
              effect: NoSchedule # This will ensure fluentd collects master logs too
          containers:
            - name: fluentd
              image: fluent/fluentd-kubernetes-daemonset:v1.12.0-debian-elasticsearch7-1.0
              env:
                - name:  FLUENT_ELASTICSEARCH_HOST
                  value: {{ .Values.elasticSearch.loggingHost | quote }}
                - name:  FLUENT_ELASTICSEARCH_PORT
                  value: {{ .Values.elasticSearch.port | quote }}
                - name: FLUENT_ELASTICSEARCH_SCHEME
                  value: {{ .Values.elasticSearch.scheme | quote }}
                #- name: FLUENT_ELASTICSEARCH_USER
                #  value: "elastic"
                #- name: FLUENT_ELASTICSEARCH_PASSWORD
                #  value: ""
                - name: K8S_NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
              resources:
                limits:
                  memory: 200Mi
                requests:
                  cpu: 100m
                  memory: 200Mi
              volumeMounts:
                - name: varlog
                  mountPath: /var/log
                - name: varlibdockercontainers
                  mountPath: /var/lib/docker/containers
                  readOnly: true
                - name: config-volume-fluentd
                  mountPath: /fluentd/etc/conf.d/
    
          terminationGracePeriodSeconds: 30
          volumes:
            - name: config-volume-fluentd
              configMap:
                name: config-volume-fluentd
                defaultMode: 0777
    
            - name: varlog
              hostPath:
                path: /var/log
            - name: varlibdockercontainers
              hostPath:
                path: /var/lib/docker/containers
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: fluentd
      namespace: kube-system
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: fluentd
    rules:
      - apiGroups:
          - ""
        resources:
          - pods
          - namespaces
        verbs:
          - get
          - list
          - watch
    
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: fluentd
    roleRef:
      kind: ClusterRole
      name: fluentd
      apiGroup: rbac.authorization.k8s.io
    subjects:
      - kind: ServiceAccount
        name: fluentd
        namespace: kube-system
    

    Config map:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: config-volume-fluentd
      namespace: kube-system
      labels:
        app: fluentd
        kubernetes.io/cluster-service: "true"
    #    addonmanager.kubernetes.io/mode: Reconcile
    data:
      fluentd.conf: |-
    
        <filter kubernetes.var.log.containers.app-resolver-**>
          @type parser
          key_name log
          reserve_data true
          remove_key_name_field false
          hash_value_field json_log
          emit_invalid_record_to_error false
          <parse>
            @type json
          </parse>
        </filter>
    
        <filter kubernetes.var.log.containers.scan-app-**>
          @type parser
          key_name log
          reserve_data true
          remove_key_name_field false
          hash_value_field json_log
          emit_invalid_record_to_error false
          <parse>
            @type json
          </parse>
        </filter>
    
        <match kubernetes.var.log.containers.app-resolver-**>
          @type elasticsearch
          host {{ .Values.elasticSearch.loggingHost }}
          port {{ .Values.elasticSearch.port }}
          scheme {{ .Values.elasticSearch.scheme }}
          log_es_400_reason true
          logstash_format true
          logstash_dateformat {{ .Values.elasticSearch.dateFormat }}
          logstash_prefix k8s-sqs-services
        </match>
    
        <match kubernetes.var.log.containers.scan-app-**>
          @type elasticsearch
          host {{ .Values.elasticSearch.loggingHost }}
          port {{ .Values.elasticSearch.port }}
          scheme {{ .Values.elasticSearch.scheme }}
          log_es_400_reason true
          logstash_format true
          logstash_dateformat {{ .Values.elasticSearch.dateFormat }}
          logstash_prefix k8s-sqs-services
        </match>
    
      containers.input.conf: |-
        <match raw.kubernetes.**>
          @id raw.kubernetes
          @type detect_exceptions
          remove_tag_prefix raw
          message log
          stream stream
          multiline_flush_interval 5
          max_bytes 500000
          max_lines 1000
        </match>
      system.conf: |-
        <system>
          root_dir /tmp/fluentd-buffers/
        </system>
      system.input.conf: |-
    
        <source>
          @id cluster-autoscaler.log
          @type tail
          format multiline
          multiline_flush_interval 5s
          format_firstline /^wd{4}/
          format1 /^(?<severity>w)(?<time>d{4} [^s]*)s+(?<pid>d+)s+(? 
        <source>
          [^ ]]+)] (?<message>.*)/
          time_format %m%d %H:%M:%S.%N
          path /var/log/cluster-autoscaler.log
          pos_file /var/log/cluster-autoscaler.log.pos
          tag cluster-autoscaler
        </source>
    
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search