CentOS Question posted in
The official documentation can be found here.

Kibana restarts every 14seconds in CentOS7 Core

This is a newly installed Elasticsearch and Kibana on CentOS7 Core. This system was hardened following “CIS_CentOS_Linux_7_Benchmark_v2.2.0.pdf”. I’m experiencing a very weird situation wherein Kibana restarts every 14 seconds, please refer to the following for session excerpts:

[root@syslog ~]#
[root@syslog ~]# cat /etc/*release
CentOS Linux release 7.6.1810 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.6.1810 (Core)
CentOS Linux release 7.6.1810 (Core)
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# rpm -qi kibana
Name        : kibana
Version     : 7.4.0
Release     : 1
Architecture: x86_64
Install Date: Tue 15 Oct 2019 04:12:07 PM +08
Group       : default
Size        : 721891086
License     : Elastic License
Signature   : RSA/SHA512, Fri 27 Sep 2019 06:42:56 PM +08, Key ID d27d666cd88e42b4
Source RPM  : kibana-7.4.0-1.src.rpm
Build Date  : Fri 27 Sep 2019 05:20:41 PM +08
Build Host  : packer-virtualbox-iso-1559162487
Relocations : /
Packager    : Kibana Team <[email protected]>
Vendor      : Elasticsearch, Inc.
URL         : https://www.elastic.co
Summary     : Explore and visualize your Elasticsearch data
Description :
Explore and visualize your Elasticsearch data
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# rpm -qi elasticsearch
Name        : elasticsearch
Epoch       : 0
Version     : 7.4.0
Release     : 1
Architecture: x86_64
Install Date: Tue 15 Oct 2019 03:58:18 PM +08
Group       : Application/Internet
Size        : 491338024
License     : Elastic License
Signature   : RSA/SHA512, Fri 27 Sep 2019 06:39:55 PM +08, Key ID d27d666cd88e42b4
Source RPM  : elasticsearch-7.4.0-1-src.rpm
Build Date  : Fri 27 Sep 2019 04:49:21 PM +08
Build Host  : packer-virtualbox-iso-1559162487
Relocations : /usr
Packager    : Elasticsearch
Vendor      : Elasticsearch
URL         : https://www.elastic.co/
Summary     : Distributed RESTful search engine built for the cloud
Description :
Reference documentation can be found at
https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
and the 'Elasticsearch: The Definitive Guide' book can be found at
https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# grep -v ^# /etc/kibana/kibana.yml | sed '/^$/d'
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://127.0.0.1:9200"]
elasticsearch.requestTimeout: 50000
elasticsearch.startupTimeout: 50000
pid.file: /var/run/kibana/kibana.pid
logging.dest: /var/log/kibana/kibana.log
logging.verbose: true
xpack.security.encryptionKey: "kibanaencryptionKey4n0th3r3ncrypt10nK3y"
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# grep -v ^# /etc/elasticsearch/elasticsearch.yml | sed '/^$/d'
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: localhost
http.port: 9200
[root@syslog ~]#
[root@syslog ~]# curl http://127.0.0.1:9200
{
  "name" : "syslog",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "lQmq1kcpTmKd5JAXrpg3-w",
  "version" : {
    "number" : "7.4.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "22e1767283e61a198cb4db791ea66e3f11ab9910",
    "build_date" : "2019-09-27T08:36:48.569419Z",
    "build_snapshot" : false,
    "lucene_version" : "8.2.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# systemctl start kibana
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 2s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 3s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 5s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 6s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 7s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 8s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 9s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 10s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 12s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 13s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 14s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           ‣ 25870 [node]

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 839ms ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 2s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 3s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 4s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 5s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 6s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 7s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 8s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 9s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 10s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 11s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 12s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:10 +08; 13s ago
 Main PID: 25906 (node)
   CGroup: /system.slice/kibana.service
           └─25906 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:10 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:24 +08; 332ms ago
 Main PID: 25943 (node)
   CGroup: /system.slice/kibana.service
           └─25943 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:24 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 1s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 2s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 3s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 4s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 5s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 6s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 7s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 8s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 9s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 10s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 11s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 12s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 13s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:39 +08; 13s ago
 Main PID: 25981 (node)
   CGroup: /system.slice/kibana.service
           └─25981 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:39 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:53 +08; 1ms ago
 Main PID: 26021 ((kibana))
   CGroup: /system.slice/kibana.service
           └─26021 (kibana)

Oct 20 08:19:53 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:19:53 +08; 764ms ago
 Main PID: 26021 (node)
   CGroup: /system.slice/kibana.service
           └─26021 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:19:53 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# rpm -qi nginx
Name        : nginx
Epoch       : 1
Version     : 1.12.2
Release     : 3.el7
Architecture: x86_64
Install Date: Tue 15 Oct 2019 05:57:15 PM +08
Group       : System Environment/Daemons
Size        : 1575034
License     : BSD
Signature   : RSA/SHA256, Fri 10 May 2019 04:20:18 PM +08, Key ID 6a2faea2352c64e5
Source RPM  : nginx-1.12.2-3.el7.src.rpm
Build Date  : Fri 10 May 2019 04:10:45 PM +08
Build Host  : buildvm-14.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://nginx.org/
Bug URL     : https://bugz.fedoraproject.org/nginx
Summary     : A high performance web server and reverse proxy server
Description :
Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 and
IMAP protocols, with a strong focus on high concurrency, performance and low
memory usage.
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# grep -v ^# /etc/nginx/nginx.conf | sed '/^$/d'
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
    worker_connections 1024;
}
http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;
    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;
        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;
        location / {
        }
        error_page 404 /404.html;
            location = /40x.html {
        }
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}
[root@syslog ~]#
[root@syslog ~]# cat /etc/nginx/conf.d/syslog.local.conf
server {
    listen 80;
    server_name syslog.local;
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.kibana;

    location / {
        proxy_pass http://127.0.0.1:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# curl http://127.0.0.1:5601
curl: (56) Recv failure: Connection reset by peer
[root@syslog ~]#
[root@syslog ~]# curl http://127.0.0.1:5601
curl: (7) Failed connect to 127.0.0.1:5601; Connection refused
[root@syslog ~]#
[root@syslog ~]# curl http://127.0.0.1:5601
Kibana server is not ready yet[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# curl http://127.0.0.1:5601
curl: (56) Recv failure: Connection reset by peer
[root@syslog ~]#
[root@syslog ~]# curl http://127.0.0.1:5601
curl: (7) Failed connect to 127.0.0.1:5601; Connection refused
[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]# curl http://127.0.0.1:5601
Kibana server is not ready yet[root@syslog ~]#
[root@syslog ~]#
[root@syslog ~]#

The kibana.log can be seen/downloaded at here

Thank you

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I re-installed Elasticsearch and Kibana but still getting that restart every 14 seconds. I was able to resolve this issue by removing "noexec" from /var; got the solution from https://discuss.elastic.co/t/kibana-server-is-not-ready-yet-throw-er-unhandled-error-event/175879/7

    I actually debugged the startup of Kibana and this was the log I got during the stop to start:

    Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["status","plugin:[email protected]","info"],"pid":14119,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"The Reporting plugin encountered issues launching Chromium in a self-test. You may have trouble generating reports: [TypeError: input.on is not a function]"}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"See Chromium's log output at "/var/lib/kibana/headless_shell-linux/chrome_debug.log""}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"Reporting plugin self-check failed. Please check the Kibana Reporting settings. Error: Could not close browser client handle!"}
Oct 30 15:58:17 syslog kibana: events.js:174
Oct 30 15:58:17 syslog kibana: throw er; // Unhandled 'error' event
Oct 30 15:58:17 syslog kibana: ^
Oct 30 15:58:17 syslog kibana: Error: spawn /var/lib/kibana/headless_shell-linux/headless_shell EACCES
Oct 30 15:58:17 syslog kibana: at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
Oct 30 15:58:17 syslog kibana: at onErrorNT (internal/child_process.js:415:16)
Oct 30 15:58:17 syslog kibana: at process._tickCallback (internal/process/next_tick.js:63:19)
Oct 30 15:58:17 syslog kibana: Emitted 'error' event at:
Oct 30 15:58:17 syslog kibana: at Process.ChildProcess._handle.onexit (internal/child_process.js:246:12)
Oct 30 15:58:17 syslog kibana: at onErrorNT (internal/child_process.js:415:16)
Oct 30 15:58:17 syslog kibana: at process._tickCallback (internal/process/next_tick.js:63:19)
Oct 30 15:58:17 syslog systemd: kibana.service: main process exited, code=exited, status=1/FAILURE
Oct 30 15:58:17 syslog systemd: Unit kibana.service entered failed state.
Oct 30 15:58:17 syslog systemd: kibana.service failed.
Oct 30 15:58:17 syslog systemd: kibana.service holdoff time over, scheduling restart.
Oct 30 15:58:17 syslog systemd: Stopped Kibana.
Oct 30 15:58:17 syslog systemd: Started Kibana.
Oct 30 15:58:20 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:20Z","tags":["info","plugins-system"],"pid":14144,"message":"Setting up [4] plugins: [security,translations,inspector,data]"}
Oct 30 15:58:20 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:20Z","tags":["info","plugins","security"],"pid":14144,"message":"Setting up plugin"}

    which led me to search for "kibana: Error: spawn /var/lib/kibana/headless_shell-linux/headless_shell EACCES" that directed me to the solution URL. Thanks to Kaldayr for the solution.


  2. 0

    One other potential place to look for info on Kibana shutting down and restarting is /var/log/messages 

    grep kibana /var/log/messages
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search