I have ECS cluster configured on EC2 instances which has a set of services running in it. I've configured AWS Firelens sidecar container to route the ECS container logs to AWS Data firehose. As per my current setup I have…
My IT team recently activated the MySQL general log to meet regulatory requirements. However, we’ve encountered significant performance issues: After 20+ hours of activation, the general log generated over 3 million rows. Today, when my team attempted to query the…
In fluentd, a source is syslog and the target is fluent-plugin-cloudwatch-logs plugin. Everything works, only the time is wrong. <source> @type syslog <transport tcp> </transport> <parse> message_format auto parser_type string </parse> tag isam </source> <match mysyslog> @type copy <store> @type…
I tried to record some logs for front-end nginx-based containers using fluentd docker logging driver, but failed. I ended up with following configuration for fluentd (located in /tmp/fluentd/fluent.conf): <source> @type forward port 24224 bind 0.0.0.0 </source> <match **-**> @type stdout…
I am having trouble sending API logs from one Fluentd server to another, and then to Elasticsearch. When ! tried sending logs directly from one Fluentd server to Elasticsearch, it worked fine. Now, I suspect there might be a problem…
I want a regular expression for using in fluentd for parsing nginx error logs. The sample row is: 2024/04/15 09:06:29 [error] 3443790#3443790: *176070165 limiting requests, excess: 2.957 by zone "RequestLimitForCommonApi", client: 77.81.151.129, server: test.com, request: "POST /capi/session/forgot HTTP/1.1", host: "test.com",…
I am trying to create a dockerfile that will have image 1 and image 2 pulled from dockerhub. It seems that second image is overriding the changes of first image. How to handle this scenario. FROM clamav/clamav:1.1 COPY clamd.conf /etc/clamav…
I have this fluentd configuration: <source> @type tail <parse> @type regexp expression /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>w+) (?<path>[^ ]*) (?<http>[^ ]*)" (?<status_code>[^ ]*) (?<size>[^ ]*)(?:s"(?<referer>[^"]*)") "(?<agent>[^"]*)" (?<urt>[^"]*).*/ time_format %d/%b/%Y:%H:%M:%S %z keep_time_key true types size:integer,reqtime:float,uct:float,uht:float,urt:float </parse> path /var/log/nginx/access.log…
I am setting up a stack with an application consisting of nginx, redis, mysql, myapp. Nginx proxies requests to myapp. I want to send logs from nginx to EFK stack, but an error occurs when starting the nginx service: Error…
I'm trying to redirect Kubernetes logs from containers to OpenSearch. But there is always some error with the date. What am I doing wrong? Docker logs example: {"log":"time="2022-04-01T10:02:31Z" level=warning msg="Cannot take snapshot backup" controller=longhorn-backup error="could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to…