Nginx Question posted in
The official documentation can be found here.

How to create grok filter logic for Nginx error log

I am trying to write Logstash filter using grok logic pattern. I am logging nginx error log. My problem is the log from the nginx contains 2 field which are not present in every error logs. I have used this logic pattern to to filter the log. I can write 2 patterns and add those 2 pattern inside the logstash filter config file. But is there any way to do it in a single logic pattern?

Logic pattern:

(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) [%{LOGLEVEL:severity}] %{POSINT:pid}#%{NUMBER:threadid}: *%{NUMBER:connectionid} %{GREEDYDATA:message}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: "%{GREEDYDATA:request}", upstream: "%{URI:upstream}", host: "%{IPORHOST:host}", referrer: "(?:%{URI:referrer})"

Sample Log 1:

2021/09/16 15:58:59 [warn] 104255#104555: *611541 an upstream response
is buffered to a temporary file
/var/cache/nginx/proxy_temp/6/06/0000000066 while reading upstream,
client: 100.16.127.2, server: webservice.example.com, request:
"GET /example/contents/common/img/somename.png
HTTP/1.1", upstream:
"https://120.27.2.87:43/example/contents/common/img/somename.png", host: "webservice.example.com", referrer:
"https://webservice.example.com/example/view/common/api_use/"

Sample Log 2:

2021/09/16 10:38:54 [error] 104555#104555: *611070 open()
"/usr/share/nginx/html/webservice.example.com/xxxxxxxx" failed
(2: No such file or directory), client: 120.36.230.2, server:
webservice.example.com, request: "GET /xxxxxxxx HTTP/1.1", host:
"webservice.example.com

As you can see, sample log 2 does not have upstream and referrer field. My logic pattern works for the logs containing upstream and referrer field. Please suggest me what should I do? Should I use 2 logic pattern inside Logstash filter config or do I need to modify my current logic pattern to match both type of logs.

Tags:

2

Answers


  1. 0

    If you issue is only that some fields are missing, then I would suggest you to not match the fields directly, but to use kv logstash filter. It let’s you define options for key-value type of structures inside of string, then it splits it and makes a map inside of event.

    You would need to capture whole key-value part of your log into one field containing all the fields and values and then set the options such as field_split_pattern to match your case and then set target to field you want the map in after kv is done parsing.

    More information here.

    In you case it would be something like this

    kv {
  field_split_pattern => ", "
  value_split_pattern => ": "
  target => "field_name"
}
    Login or Signup to reply.
  2. 0

    You can manage this 2 sample in the grok expression, just saif upstream and referrer are optionnal like this :

    [, referrer: "(?:%{URI:referrer})"]?

    So finally the full line which i’ve tested on grokdebug tools and it works for your 2 logs sample :

    %{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME} [%{LOGLEVEL:severity}] %{POSINT:pid}#%{NUMBER:threadid}: *%{NUMBER:connectionid} %{GREEDYDATA:message}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: "%{GREEDYDATA:request}", [upstream: "%{URI:upstream}", ]?host: "%{IPORHOST:host}"[, referrer: "(?:%{URI:referrer})"]?
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search