skip to Main Content

I’m trying to make filebeat send log to logstash on another machine and I just can’t get it to work. This is the filebeat.yml configuration:

https://pastebin.com/8a2RtGBa (Using pastebin because of character limit)

This is the configuration on the machine that has logstash:

[root@ELK ~]$ cat /etc/logstash/conf.d/testing.conf 
input{
file{
path => "/var/log/commands.log"
}
beats{
port => 5044
}
}
filter {
if [path] == "/var/log/commands.log" {
grok{
match => { "message" => "[(%{TIMESTAMP_ISO8601:sys_timestamp})]s(?<field1>[0-9a-zA-Z_-]+)s(?<field2>[0-9a-zA-Z_-]+):USER=(?<field3>[0-9a-zA-Z_-]+)sPWD=(?<field4>[0-9a-zA-Z_/-]+)sPID=[(?<field5>[0-9]+)]sCMD="(?<field6>.*)"sExit=[(?<field7>[0-9]+)]sCONNECTION=(?<field8>.*)"
}
}
}
}
output{
elasticsearch { hosts => ["localhost:9200"]}
}

This is what I get if I try setting filebeat with "filebeat setup -e":

[root@server150 ~]# filebeat setup -e
2020-07-17T08:12:36.199Z        INFO    instance/beat.go:647    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-07-17T08:12:36.200Z        INFO    instance/beat.go:655    Beat ID: aa84fd5b-d016-4688-a4a1-172dbcf2054a
2020-07-17T08:12:36.202Z        INFO    [beat]  instance/beat.go:983    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "aa84fd5b-d016-4688-a4a1-172dbcf2054a"}}}
2020-07-17T08:12:36.203Z        INFO    [beat]  instance/beat.go:992    Build info      {"system_info": {"build": {"commit": "f79387d32717d79f689d94fda1ec80b2cf285d30", "libbeat": "7.8.0", "time": "2020-06-14T18:15:37.000Z", "version": "7.8.0"}}}
2020-07-17T08:12:36.203Z        INFO    [beat]  instance/beat.go:995    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.10"}}}
2020-07-17T08:12:36.204Z        INFO    [beat]  instance/beat.go:999    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-09T11:44:16Z","containerized":false,"name":"server150","ip":["127.0.0.1/8","*ip-of-thismachine*/25"],"kernel_version":"3.10.0-1062.18.1.el7.x86_64","mac":["00:0c:29:e5:a1:fa"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"3eec437c66d444a59ef5f075a429441d"}}}
2020-07-17T08:12:36.204Z        INFO    [beat]  instance/beat.go:1028   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 12559, "ppid": 11800, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-07-17T08:12:35.340Z"}}}
2020-07-17T08:12:36.204Z        INFO    instance/beat.go:310    Setup Beat: filebeat; Version: 7.8.0
2020-07-17T08:12:36.205Z        INFO    [publisher]     pipeline/module.go:113  Beat name: server150
2020-07-17T08:12:36.206Z        WARN    beater/filebeat.go:156  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-07-17T08:12:36.206Z        ERROR   instance/beat.go:958    Exiting: Index management requested but the Elasticsearch output is not configured/enabled
Exiting: Index management requested but the Elasticsearch output is not configured/enabled

This is what I get when I try to run filebeat with "filebeat -e":

[root@server150 ~]# filebeat -e
2020-07-17T08:16:47.104Z        INFO    instance/beat.go:647    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-07-17T08:16:47.104Z        INFO    instance/beat.go:655    Beat ID: aa84fd5b-d016-4688-a4a1-172dbcf2054a
2020-07-17T08:16:47.107Z        INFO    instance/beat.go:404    filebeat stopped.
2020-07-17T08:16:47.107Z        ERROR   instance/beat.go:958    Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

Does anyone knows what’s going on? I just can’t get it to work. Huge thanks ahead!

2

Answers


  1. go to ur yml file of your beat , and comment :

    #setup.template.settings:
       #index.number_of_shards: 1
    

    then do :

    filebeat setup -e
    

    ( same work for different beats)
    you will receive finally :
    Kibana dashboards successfully loaded. Loaded dashboards

    Login or Signup to reply.
  2. I recommend you my configs.

    process : *.log -> filebeat -> logstash -> elasticsearch

    filebeat.yml :

    filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /usr/share/filebeat/mylog/*.log
    
    output.logstash:
      hosts: ["logstash:5044"]
    

    logstash.conf :

    input {
      beats {
        port => 5044
      }
    }
    
    filter {
      json {
        source => "message"
      }
      date{
        match => ["timestamp", "UNIX_MS"]
        target => "@timestamp"
      }
      ruby {
        code => "event.set('indexDay', event.get('[@timestamp]').time.localtime('+00:00').strftime('%Y%m%d'))"
      }
    }
    
    output {
      elasticsearch {
        hosts => ["elasticsearch:9200"]
        template => "/usr/share/logstash/templates/logstash.template.json"
        template_name => "logstash"
        template_overwrite => true
        index => "logstash-%{indexDay}"
        codec => json
      }
      stdout {
        codec => rubydebug
      }
    }
    

    logstash.template.json:

    {
        "template": "logstash",
        "index_patterns": [
            "logstash-*"
        ],
        "settings": {
            "number_of_shards": 1
        },
        "mappings": {
            "properties": {
                "name": {
                    "type": "keyword"
                },
                "class": {
                    "type": "keyword"
                },
                "state": {
                    "type": "integer"
                },
                "@timestamp": {
                    "type": "date"
                }
            }
        }
    }
    

    You can find a simple solution in below github repository.

    docker-compose config : ELK with filebeat by docker-compose

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search