I am a new DevOps Engineer. I was checking out our DEV AKS cluster at work and noticed that Fluentd is using a crazy amount of memory and isn't releasing it back, example below: fluentd-dev-95qmh 13m 1719Mi fluentd-dev-fhd4w 9m 1732Mi…
I want to send some nginx logs from fluentd to elasticsearch , however, fluentd is unable to start due to following error message: The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible…
I am using AWS Opensearch to retrieve the logs from all my Kubernetes applications. I have the following pods: Kube-proxy, Fluent-bit, aws-node, aws-load-balancer-controller, and all my apps (around 10). While fluent-bit successfully send all the logs from Kube-proxy, Fluent-bit, aws-node…
I want to build the efk logger system by docker compose. Everything is setup, only fluentd has problem. fluentd docker container logs 2022-02-15 02:06:11 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf" 2022-02-15 02:06:11 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.3'…
I have created a docker file : FROM fluentd:v1.14.0-debian-1.0 USER root RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.1.2"] USER fluent From which I create a container in my compose file: version: '3' services: fluentd: build: ./fluentd volumes: - ./fluentd/conf:/fluentd/etc links:…
We are using AWS Elasticsearch Service and fleuntd to push the logs of microservice. We have installed fluentd on EC2 instance using docker based configuration. We have followed the steps mentioned in the https://docs.fluentd.org/container-deployment/docker-compose and also it was working pretty…
I'm using that fluentd daemonset docker image and sending logs to ES with fluentd is working perfectly by the way of using following code-snippets: containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:v1.4.2-debian-elasticsearch-1.1 env: - name: FLUENT_ELASTICSEARCH_HOST value: "my-aws-es-endpoint" - name: FLUENT_ELASTICSEARCH_PORT value:…
So I whipped up a docker-based fluentd TCP log collector. Following the examples here, https://docs.fluentd.org/input/tcp , led to successfully sending a line from my host Win 10 WSL (Debian) by saying echo "my_service: 08:03:10 INFO [my_py_file:343]: My valuable log info."…
I wish to process (call a parser on) the contents of a log event via fluend. These come different formats (json, apache2, nginx and bespoke regexp), sometime I may wish to skip any processing. Events come from the same source…
I have problem with using env at td-agent config, I tried: <source> @type tail path /home/td-agent/test.txt tag "#{ENV['WEBTEST']}" pos_file /var/log/td-agent/td-agent-test.pos @include /etc/td-agent/web_parse_regex.conf </source> /etc/sysconfig/td-agent : export WEBTEST="webtest" and when I start td-agent and check td-agent.log, tag is empty 2020-06-09 15:40:20…