skip to Main Content

Going through the Elasticsearch docs for setting up Elasticsearch/Kibana with Docker, but I’m getting several errors. I follow the steps exactly. I’m running this on an Ubuntu 20.04 EC2 instance. What am I doing wrong?

Here’s what I did:

  1. docker pull docker.elastic.co/elasticsearch/elasticsearch:8.0.0
  2. docker pull docker.elastic.co/kibana/kibana:8.0.0
  3. docker network create elastic
  4. docker run --name es01 --net elastic -p 9200:9200 -it docker.elastic.co/elasticsearch/elasticsearch:8.0.0

After step 4, Elasticsearch says:

A password is generated for the elastic user and output to the terminal, plus enrollment tokens for enrolling Kibana and adding additional nodes to your cluster.

I get neither. Instead, I get these error logs:

{"@timestamp":"2022-02-24T22:28:24.318Z", "log.level":"ERROR", "message":"Failed to create enrollment token when generating API key", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[a178a1e9d98b][generic][T#4]","log.logger":"org.elasticsearch.xpack.security.enrollment.InternalEnrollmentTokenGenerator","elasticsearch.cluster.uuid":"mC4ceJ2nT7i9-QF6V537Zg","elasticsearch.node.id":"IthIKocaSACunHatZxePPw","elasticsearch.node.name":"a178a1e9d98b","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException","error.message":"[.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][fobYLX8BZdXU5J2_mb_p], source[{"doc_type":"api_key","creation_time":1645741644265,"expiration_time":1645743444265,"api_key_invalidated":false,"api_key_hash":"{PBKDF2}10000$PbPNTKm9i5HBuHO+W9snM/+0C1sf4OGjE3xC1m3xKew=$oQXD/UOSgR/hDNHz1IgNKoVOG4Zi0LkiPQW3IMPnRtA=","role_descriptors":{"create_enrollment_token":{"cluster":["cluster:admin/xpack/security/enroll/node"],"indices":[],"applications":[],"run_as":[],"metadata":{},"type":"role"}},"limited_by_role_descriptors":{"superuser":{"cluster":["all"],"indices":[{"names":["*"],"privileges":["all"],"allow_restricted_indices":false},{"names":["*"],"privileges":["monitor","read","view_index_metadata","read_cross_cluster"],"allow_restricted_indices":true}],"applications":[{"application":"*","privileges":["*"],"resources":["*"]}],"run_as":["*"],"metadata":{"_reserved":true},"type":"role"}},"name":"enrollment_token_API_key_fYbYLX8BZdXU5J2_mb_p","version":8000099,"metadata_flattened":null,"creator":{"principal":"_xpack_security","full_name":null,"email":null,"metadata":{},"realm":"__attach","realm_type":"__attach"}}]}] blocking until refresh]","error.stack_trace":"org.elasticsearch.action.UnavailableShardsException: [.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][fobYLX8BZdXU5J2_mb_p], source[{"doc_type":"api_key","creation_time":1645741644265,"expiration_time":1645743444265,"api_key_invalidated":false,"api_key_hash":"{PBKDF2}10000$PbPNTKm9i5HBuHO+W9snM/+0C1sf4OGjE3xC1m3xKew=$oQXD/UOSgR/hDNHz1IgNKoVOG4Zi0LkiPQW3IMPnRtA=","role_descriptors":{"create_enrollment_token":{"cluster":["cluster:admin/xpack/security/enroll/node"],"indices":[],"applications":[],"run_as":[],"metadata":{},"type":"role"}},"limited_by_role_descriptors":{"superuser":{"cluster":["all"],"indices":[{"names":["*"],"privileges":["all"],"allow_restricted_indices":false},{"names":["*"],"privileges":["monitor","read","view_index_metadata","read_cross_cluster"],"allow_restricted_indices":true}],"applications":[{"application":"*","privileges":["*"],"resources":["*"]}],"run_as":["*"],"metadata":{"_reserved":true},"type":"role"}},"name":"enrollment_token_API_key_fYbYLX8BZdXU5J2_mb_p","version":8000099,"metadata_flattened":null,"creator":{"principal":"_xpack_security","full_name":null,"email":null,"metadata":{},"realm":"__attach","realm_type":"__attach"}}]}] blocking until refresh]ntat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1076)ntat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:872)ntat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)ntat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1031)ntat org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)ntat org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)ntat org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:651)ntat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)ntat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)ntat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)ntat java.base/java.lang.Thread.run(Thread.java:833)n"}
{"@timestamp":"2022-02-24T22:28:47.612Z", "log.level":"ERROR", "message":"error downloading geoip database [GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[a178a1e9d98b][generic][T#6]","log.logger":"org.elasticsearch.ingest.geoip.GeoIpDownloader","elasticsearch.cluster.uuid":"mC4ceJ2nT7i9-QF6V537Zg","elasticsearch.node.id":"IthIKocaSACunHatZxePPw","elasticsearch.node.name":"a178a1e9d98b","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException","error.message":"[.geoip_databases][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.geoip_databases][0]] containing [index {[.geoip_databases][GeoLite2-ASN.mmdb_0_1645741637264], source[n/a, actual length: [1mb], max length: 2kb]}]]","error.stack_trace":"org.elasticsearch.action.UnavailableShardsException: [.geoip_databases][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.geoip_databases][0]] containing [index {[.geoip_databases][GeoLite2-ASN.mmdb_0_1645741637264], source[n/a, actual length: [1mb], max length: 2kb]}]]ntat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1076)ntat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:872)ntat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)ntat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1031)ntat org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)ntat org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)ntat org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:651)ntat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)ntat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)ntat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)ntat java.base/java.lang.Thread.run(Thread.java:833)n"}
{"@timestamp":"2022-02-24T22:28:54.310Z", "log.level":"ERROR", "message":"Failed to generate credentials for the elastic built-in superuser", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[a178a1e9d98b][generic][T#7]","log.logger":"org.elasticsearch.xpack.security.InitialNodeSecurityAutoConfiguration","elasticsearch.cluster.uuid":"mC4ceJ2nT7i9-QF6V537Zg","elasticsearch.node.id":"IthIKocaSACunHatZxePPw","elasticsearch.node.name":"a178a1e9d98b","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException","error.message":"[.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][reserved-user-elastic], source[{"password":"ff1DWkSBw4Cju0b8U7PM","enabled":true,"type":"reserved-user"}]}] and a refresh]","error.stack_trace":"org.elasticsearch.action.UnavailableShardsException: [.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][reserved-user-elastic], source[{"password":"ff1DWkSBw4Cju0b8U7PM","enabled":true,"type":"reserved-user"}]}] and a refresh]ntat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1076)ntat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:872)ntat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)ntat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1031)ntat org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)ntat org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)ntat org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:651)ntat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)ntat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)ntat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)ntat java.base/java.lang.Thread.run(Thread.java:833)n"}

They are long messages. Here’s the error portion of the above:

"Failed to create enrollment token when generating API key"
"error downloading geoip database [GeoLite2-ASN.mmdb]"
"Failed to generate credentials for the elastic built-in superuser"
"error downloading geoip database [GeoLite2-City.mmdb]"

2

Answers


  1. Chosen as BEST ANSWER

    I did not have enough storage space.

    A gentleman on the Elasticsearch Slack channel was kind enough to point out that this was the real culprit:
    "error.type": "org.elasticsearch.action.UnavailableShardsException", "error.message": "[.security-7][0] primary shard is not active Timeout: [1m],

    I looked at my available host system storage space and found there was only 17G available! Cleaning up my Trash bin fixed the issue. Works now. Hopefully this helps someone else!


  2. I assume your problem is because of the network, since you got failed when attempting to downloading geoip database and you use docker to run it.
    https://www.elastic.co/blog/docker-networking

    When running Elasticsearch, you will need to ensure it publishes to an
    IP address that is reachable from outside the container; this can be
    configured via the setting network.publish_host.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search