We are using AWS Elasticsearch Service and fleuntd to push the logs of microservice. We have installed fluentd on EC2 instance using docker based configuration. We have followed the steps mentioned in the https://docs.fluentd.org/container-deployment/docker-compose and also it was working pretty fine till last week. There is no change in the configuration of both elastcicsearch service and ec2 instance we are provisioning. Below is the error logs we are getting.
2021-08-05 20:35:57 +0000 [error]: #0 unexpected error
error_class=NoMethodError error="undefined method
host_unreachable_exceptions' for #<Elasticsearch::Transport::Client:0x00007fe5231b5e08>" 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:41:in
rescue in retry_operate’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:39:in
retry_operate' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:487:in
handle_last_seen_es_major_version’ 2021-08-05 20:35:57 +0000
[error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:339:in
configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in
configure’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:132:in
add_match' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in
block in configure’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in
each' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in
configure’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:146:in
configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in
configure’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in
run_configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:714:in
block in run_worker’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:966:in
main_process' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:706:in
run_worker’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:364:in
<top (required)>' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in
require’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in
require' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in
<top
(required)>’ 2021-08-05 20:35:57 +0000 [error]: #0
/usr/bin/fluentd:23:inload' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/bin/fluentd:23:in
‘
Below is the configuration used.
docker-compose.yml
version: '3'
services:
fluentd:
build: ./fluentd
volumes:
- ./fluentd/conf:/fluentd/etc
ports:
- "8880:8880"
- "8880:8880/udp"
Dockerfile
FROM fluent/fluentd:v1.13.3
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.5"]
USER fluent
fluentd.conf
<source>
@type forward
port 8880
bind 0.0.0.0
</source>
<match **>
host ES_HOT
port 443
scheme https
logstash_format true
logstash_prefix "fluentd-logs"
logstash_dateformat "%Y%m%d"
user "elastic"
password xxxxxx
flush_interval 10s
fail_on_detecting_es_version_retry_exceed false
verify_es_version_at_startup false
max_retry_get_es_version 2
default_elasticsearch_version 7.10
request_timeout 60s
reload_on_failure true
</match>
Our configuration with AWS Elasticsearch Service 7.10, Fluentd v1.12.0-debian-1.0 and fluent-plugin-elasticsearch 4.3.3, was working perfectly fine till last week. One of the ec2 instance which was provisioned 10 days back, was able to connect and push the logs as well. But when we restarted that instance its also stopped working with the same error. We have Terraform scripts that provision all infrastructure. Later, we have tried different versions combinations as well. But nothing worked out.
Thanks in advance
2
Answers
I had a similar issue. Digging into the matter, I found that the
fluent-plugin-elasticsearch
plugin doesn’t version-lock its dependencies, includingelasticsearch-transport
andelasticsearch-api
, and that a recent update somewhere in one of those led to some kind of incompatibility.The fix that worked for me was to explicitly install the dependencies at the latest issue-free version to avoid having the
fluent-plugin-elasticsearch
install pull the latest issue-causing version.Proposed Modification to your Dockerfile
I launched an AWS Elasticsearch 7.10 managed service cluster and connected an EKS cluster to it with this Fluentd docker image:
fluent/fluentd-kubernetes-daemonset:v1.12.0-debian-elasticsearch7-1.0
Deploy manifest:
Config map: