seo Question posted in
Some basic information to get you started with this can be found here.

Incorrect timestamp by date processor in elasticsearch filebeat – SEO

I am getting wrong value in @timestamp field for elasticsearch/filebeat.
My filebeat pipeline definition

    curl -H 'Content-Type: application/json' -XPUT "logger:9200/_ingest/pipeline/app_log" -d'
    {
      "description" : "Ingest pipeline for Jetty server log",
      "processors" : [
        {
          "grok": {
            "field": "message",
            "patterns": ["%{TIMESTAMP_ISO8601:timestamp} (%{UUID:accessid})? \[(?<threadname>[^\]]+)\] %{LOGLEVEL:level} %{DATA:classname} - %{GREEDYDATA:message}"]
          }
        },
        {
          "date": {
            "field": "timestamp",
            "formats": [ "yyyy-mm-dd H:m:s,SSS" ]
          }
        }
      ],
      "on_failure" : [{
        "set" : {
          "field" : "error.message",
          "value" : "{{ _ingest.on_failure_message }}"
        }
      }]
    }'

Simulation result for some sample logger line. ( Using logback configuration ) 

curl -H 'Content-Type: application/json' -XPOST "logger:9200/_ingest/pipeline/app_log/_simulate?pretty" -d'
{
  "docs": [
    {
      "_source": {
        "message": "2018-03-17 22:38:39,079 bab3157d-a11c-4dba-a6d6-c47ae0de2b7f [qtp224100622-174782] INFO  i.n.core.services.cache.CacheBuilder - Key : ChIJrTTTJkdsrjsRXkrYRKRRfd8-seo-localitiesv1 is returned from cache"

      }
    },
    {
      "_source": {
        "message": "2017-12-12 01:14:12,079  [qtp224100622-185269] WARN  i.n.m.cache.sdk.RedisCacheProvider - No matching policy: class in.nobroker.core.domain.Token"
      }
    }
  ]
}'

The result from this simulation:

{ “docs” : [
{
“doc” : {
“_index” : “_index”,
“_type” : “_type”,
“_id” : “_id”,
“_source” : {
“accessid” : “bab3157d-a11c-4dba-a6d6-c47ae0de2b7f”,
“@timestamp” : “2018-01-17T22:38:39.079Z”,
“classname” : ” i.n.core.services.cache.CacheBuilder”,
“level” : “INFO”,
“message” : “Key : ChIJrTTTJkdsrjsRXkrYRKRRfd8-seo-localitiesv1 is returned from cache”,
“timestamp” : “2018-03-17 22:38:39,079”,
“threadname” : “qtp224100622-174782”
},
“_ingest” : {
“timestamp” : “2018-03-17T15:35:35.543Z”
}
}
},
{
“doc” : {
“_index” : “_index”,
“_type” : “_type”,
“_id” : “_id”,
“_source” : {
“@timestamp” : “2017-01-12T01:14:12.079Z”,
“classname” : ” i.n.m.cache.sdk.RedisCacheProvider”,
“level” : “WARN”,
“message” : “No matching policy: class in.nobroker.core.domain.Token”,
“timestamp” : “2017-12-12 01:14:12,079”,
“threadname” : “qtp224100622-185269”
},
“_ingest” : {
“timestamp” : “2018-03-17T15:35:35.543Z”
}
}
} ] }

Please notice that @timestamp field is totally different from timestamp field.

Tags:

2

Answers


  1. 0

    Seems that you give wrong date formats

    {
  "date": {
    "field": "timestamp",
    "formats": [ "yyyy-mm-dd H:m:s,SSS" ]
  }
}

    As the doc shows “mm: two-digit minutes, zero-padded if needed” instead of the month you expected.
    Try

    {
  "date": {
    "field": "timestamp",
    "formats": [ "yyyy-MM-dd H:m:s,SSS" ]
  }
}
    Login or Signup to reply.
  2. 0

    Filebeat do not have date processor. Elasticsearch has processor. To use the timestamp from the log as @timestamp in filebeat use ingest pipeline in Elasticsearch.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search