skip to Main Content

I have a backend production with microservice architecture that I wrote with nestjs. All services here are running as separate docker containers. In addition to these services, in my docker-compose.yml file:

   elasticsearch:
     image: elasticsearch:7.6.2
     container_name: elasticsearch
     ports:
       - '9200:9200'
       - '9300:9300'
     environment:
       - cluster.name=docker-cluster
       - node.name=node-1
       - discovery.type=single-node
   logstash:
     labels:
       com.example.service: 'logstash'
       com.example.description: 'For logging data'
     image: logstash:7.6.2
     container_name: logstash
     ports:
       - '9600:9600'
     depends_on:
       - elasticsearch
     volumes:
       - '/var/lib/docker/containers:/var/lib/docker/containers:ro'

I added the parts and when I boot the project both elastichsearch and logstash start properly. then inside the log stash container
docker exec -it logstash bash
I entered with the command and configured the logstash.conf file in the pipline folder as follows.

input {
   file {
     path => "/var/lib/docker/containers/*/*-json.log"
     type => "docker"
   }
}

filter {
   if [type] == "docker" {
     grok {
       match => { "source" => "/var/lib/docker/containers/%{DATA:container_id}/*-json.log" }
     }
   }
}

output {
   elasticsearch {
     hosts => ["elasticsearch:9200"]
     index => "container-logs-%{container_id}"
   }
}

then by logging out of the container, I ran the container again with the following command

docker restart logstash

The container is started properly, but when I run the command below, I observe that no logs are indexed in elastic search. Where could this be the reason? Where do you think I am doing wrong?

curl -XGET "http://localhost:9200/_search"

I get the following result as a result of the above command.

{"took":0,"timed_out":false,"_shards":{"total":0,"successful":0,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0.0,"hits":[]}}

my logstash container logs

    [2022-12-15T11:21:14,554][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2022-12-15T11:21:14,561][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2022-12-15T11:21:14,572][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] ES Output version determined {:es_version=>7}
[2022-12-15T11:21:14,573][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-12-15T11:21:14,615][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://elasticsearch:9200"]}
[2022-12-15T11:21:14,626][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x1e9a9f6f run>"}
[2022-12-15T11:21:14,696][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2022-12-15T11:21:14,705][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
[2022-12-15T11:21:14,934][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

my elasticsearch container log

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2022-12-14T14:04:56,680Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "using [1] data paths, mounts [[/ (overlay)]], net usable_space [5.8gb], net total_space [19.2gb], types [overlay]" }
{"type": "server", "timestamp": "2022-12-14T14:04:56,684Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "heap size [1007.3mb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2022-12-14T14:04:56,759Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "node name [node-1], node ID [KSEf_YAqQwuCymU27MN1cA], cluster name [docker-cluster]" }
{"type": "server", "timestamp": "2022-12-14T14:04:56,760Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "version[7.6.2], pid[1], build[default/docker/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/5.15.0-1011-aws/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.2/13.0.2+8]" }
{"type": "server", "timestamp": "2022-12-14T14:04:56,761Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
{"type": "server", "timestamp": "2022-12-14T14:04:56,761Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-10762875107622396116, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,909Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,910Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,910Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [flattened]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,911Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [frozen-indices]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,911Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,912Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,918Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,919Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,919Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,919Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,919Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,920Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,920Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,921Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,921Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,922Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,922Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [search-business-rules]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,922Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [spatial]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,923Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [transform]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,924Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,925Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [vectors]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,925Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-analytics]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,925Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,926Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,926Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,926Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-enrich]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,926Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,927Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,927Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,927Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,927Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,928Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,929Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,929Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,929Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-voting-only-node]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,930Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,930Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2022-12-14T14:05:06,042Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2022-12-14T14:05:07,153Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "[controller/103] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV" }
{"type": "server", "timestamp": "2022-12-14T14:05:08,342Z", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security" }
{"type": "server", "timestamp": "2022-12-14T14:05:08,546Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,003Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "initialized" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,004Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "starting ..." }
{"type": "server", "timestamp": "2022-12-14T14:05:10,184Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "publish_address {192.168.0.16:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,570Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "setting initial configuration to VotingConfiguration{KSEf_YAqQwuCymU27MN1cA}" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,823Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "elected-as-master ([1] nodes joined)[{node-1}{KSEf_YAqQwuCymU27MN1cA}{7Qd6fK-7TuKA0JyF45eWHA}{192.168.0.16}{192.168.0.16:9300}{dilm}{ml.machine_memory=8230952960, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{node-1}{KSEf_YAqQwuCymU27MN1cA}{7Qd6fK-7TuKA0JyF45eWHA}{192.168.0.16}{192.168.0.16:9300}{dilm}{ml.machine_memory=8230952960, xpack.installed=true, ml.max_open_jobs=20}]}" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,892Z", "level": "INFO", "component": "o.e.c.c.CoordinationState", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "cluster UUID set to [8_YjSvrMR_Gt53WnrnviLA]" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,988Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "master node changed {previous [], current [{node-1}{KSEf_YAqQwuCymU27MN1cA}{7Qd6fK-7TuKA0JyF45eWHA}{192.168.0.16}{192.168.0.16:9300}{dilm}{ml.machine_memory=8230952960, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}" }
{"type": "server", "timestamp": "2022-12-14T14:05:11,104Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "publish_address {192.168.0.16:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:11,105Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "started", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:11,279Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "recovered [0] indices into cluster_state", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:11,671Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.triggered_watches] for index patterns [.triggered_watches*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:11,753Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.watches] for index patterns [.watches*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:11,879Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.watch-history-10] for index patterns [.watcher-history-10*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:11,965Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [ilm-history] for index patterns [ilm-history-1*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,033Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.slm-history] for index patterns [.slm-history-1*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,134Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,245Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,317Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,377Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,441Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,523Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding index lifecycle policy [watch-history-ilm-policy]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,580Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding index lifecycle policy [ilm-history-ilm-policy]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,636Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding index lifecycle policy [slm-history-ilm-policy]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,794Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "license [3693ddfc-37ba-4874-bb54-0651c5976db2] mode [basic] - valid", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T14:05:12,796Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-14T15:28:59,574Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [logstash] for index patterns [logstash-*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-15T01:30:00,014Z", "level": "INFO", "component": "o.e.x.s.SnapshotRetentionTask", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "starting SLM retention snapshot cleanup task", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-15T01:38:00,002Z", "level": "INFO", "component": "o.e.x.m.MlDailyMaintenanceService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "triggering scheduled [ML] maintenance tasks", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-15T01:38:00,004Z", "level": "INFO", "component": "o.e.x.m.a.TransportDeleteExpiredDataAction", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "Deleting expired data", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-15T01:38:00,058Z", "level": "INFO", "component": "o.e.x.m.a.TransportDeleteExpiredDataAction", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "Completed deletion of expired ML data", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }
{"type": "server", "timestamp": "2022-12-15T01:38:00,060Z", "level": "INFO", "component": "o.e.x.m.MlDailyMaintenanceService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "Successfully completed [ML] maintenance tasks", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA"  }

2

Answers


  1. You can run the following command and check the status of Logstash:

    curl -XGET 'localhost:9600/_node/stats/events?pretty'
    

    The output will be like this:

    enter image description here
    In the screenshot above my elasticsearch is not reachable so all events get stuck.

    The error I get from logstash:

    [2022-12-15T18:28:28,541][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused"}
    
    Login or Signup to reply.
  2. After trying to replicate your setup, it seems to me that it is a permission problem. Even though you are bind mounting with :r, host permissions persist into the container fs. Logstash runs with user 1000 and container logs require root permissions.

    FROM logstash:7.6.4
    USER root
    

    docker build -t mylogstash .

    Now get in and try your configs:

    logstash:
     labels:
       com.example.service: 'logstash'
       com.example.description: 'For logging data'
     image: mylogstash
     container_name: logstash
     ports:
       - '9600:9600'
     depends_on:
       - elasticsearch
     volumes:
       - '/var/lib/docker/containers:/var/lib/docker/containers:ro'
    

    I personally would not use logstash this way, instead setup Filebeat to monitor the logs with docker input or docker autodiscover. I use logstash then to parse the log message into json fields, remove unnecessary meta etc.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search