skip to Main Content

I have this fluentd configuration:

<source>
   @type tail
   <parse>
   @type regexp
    expression /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>w+) (?<path>[^ ]*) (?<http>[^ ]*)" (?<status_code>[^ ]*) (?<size>[^ ]*)(?:s"(?<referer>[^"]*)") "(?<agent>[^"]*)" (?<urt>[^"]*).*/
      time_format %d/%b/%Y:%H:%M:%S %z
      keep_time_key true
      types size:integer,reqtime:float,uct:float,uht:float,urt:float
   </parse>
   path /var/log/nginx/access.log
   pos_file /tmp/fluent_nginx.pos
   tag nginx
</source>

My log format:

193.137.78.17 - - [07/Jan/2023:09:21:59 +0000] "GET /net/api/employee HTTP/1.1" 200 2323 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 0.014
193.137.78.17 - - [07/Jan/2023:09:22:00 +0000] "GET /net/api/employee HTTP/1.1" 200 2323 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 0.005

I’ve tested my regex on regex101 and works without problems. Still, I get a no patterns matched warning on fluentd. I don’t understand why the log isn’t parsed correctly.

Jan 07 09:26:26 srv-api fluentd[14878]: 2023-01-07 09:26:26 +0000 [warn]: #0 no patterns matched tag="nginx"

Can anyone help me, please? Thanks!

2

Answers


  1. I think your problem is leading spaces in the log

    Your pattern is insisting that the <remote> has no spaces before it, but you do have 4 spaces in your log before the remote IP.

    The simplest way, to my mind, is to insert an optional variable-number-of-spaces at the beginning.

    ^( )*(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>w+) (?<path>[^ ]*) (?<http>[^ ]*)" (?<status_code>[^ ]*) (?<size>[^ ]*)(?:s"(?<referer>[^"]*)") "(?<agent>[^"]*)" (?<urt>[^"]*).*
    

    How it works

    The ( and ) are just to make life easier for people reading the code: they will see that between them is a space character, which they might not otherwise notice.

    The * means 0 or more of these.

    This allows 0 or more spaces at the beginning of the line to be matched and discarded.

    Incidentally

    I noticed you are sometimes escaping " with and sometimes not. Is there a reason for this?

    Login or Signup to reply.
  2. You should directly use the nginx parser plugin instead.

    Here is a complete working example with the sample input plugin and the nginx parser plugin:

    fluent-nginx-test.conf

    <source>
      @type sample
      sample [
        { "message": "193.137.78.17 - - [07/Jan/2023:09:22:00 +0000] "GET /net/api/employee HTTP/1.1" 200 2323 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 0.005" },
        { "message": "193.137.78.18 - - [07/Jan/2023:09:22:00 +0000] "GET /net/api/employee HTTP/1.1" 200 2323 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 0.005" }
      ]
      rate 1
      size 2
      tag nginx
    </source>
    
    <filter nginx>
      @type parser
      key_name message
      <parse>
        @type nginx
      </parse>
    </filter>
    
    <match nginx>
      @type stdout
    </match>
    

    Run

    $ fluentd -c ./fluent-nginx-test.conf
    

    Output

    2023-01-07 14:22:00.000000000 +0500 nginx: {"remote":"193.137.78.17","host":"-","user":"-","method":"GET","path":"/net/api/employee","code":"200","size":"2323","referer":"-","agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36","http_x_forwarded_for":"0.005"}
    2023-01-07 14:22:00.000000000 +0500 nginx: {"remote":"193.137.78.18","host":"-","user":"-","method":"GET","path":"/net/api/employee","code":"200","size":"2323","referer":"-","agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36","http_x_forwarded_for":"0.005"}
    

    Apart from that, I used your regex with the regexp parser plugin and it’s working fine too (though there are redundant values in types field):

    fluent-nginx-test-with-regexp.conf

    <source>
      @type sample
      sample [
        { "message": "193.137.78.17 - - [07/Jan/2023:09:22:00 +0000] "GET /net/api/employee HTTP/1.1" 200 2323 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 0.005" },
        { "message": "193.137.78.18 - - [07/Jan/2023:09:22:00 +0000] "GET /net/api/employee HTTP/1.1" 200 2323 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 0.005" }
      ]
      rate 1
      size 2
      tag nginx
    </source>
    
    <filter nginx>
      @type parser
      key_name message
      <parse>
        @type regexp
        expression /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>w+) (?<path>[^ ]*) (?<http>[^ ]*)" (?<status_code>[^ ]*) (?<size>[^ ]*)(?:s"(?<referer>[^"]*)") "(?<agent>[^"]*)" (?<urt>[^"]*).*/
        time_format %d/%b/%Y:%H:%M:%S %z
        keep_time_key true
        types size:integer,reqtime:float,uct:float,uht:float,urt:float
       </parse>
    </filter>
    
    <match nginx>
      @type stdout
    </match>
    

    Run

    $ fluentd -c ./fluent-nginx-test-with-regexp.conf
    

    Output

    2023-01-07 14:22:00.000000000 +0500 nginx: {"remote":"193.137.78.17","host":"-","user":"-","time":"07/Jan/2023:09:22:00 +0000","method":"GET","path":"/net/api/employee","http":"HTTP/1.1","status_code":"200","size":2323,"referer":"-","agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36","urt":0.005}
    2023-01-07 14:22:00.000000000 +0500 nginx: {"remote":"193.137.78.18","host":"-","user":"-","time":"07/Jan/2023:09:22:00 +0000","method":"GET","path":"/net/api/employee","http":"HTTP/1.1","status_code":"200","size":2323,"referer":"-","agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36","urt":0.005}
    

    However, the error no patterns matched tag="nginx" in message:

    Jan 07 09:26:26 srv-api fluentd[14878]: 2023-01-07 09:26:26 +0000 [warn]: #0 no patterns matched tag="nginx"
    

    which means that you don’t have a corresponding match section in your config file. You must have a match section with the respective tag that you want to process or output.

    Example:

    <source>
      @type tail
      # ...
      tag nginx
    </source>
    
    # ...
    
    <match nginx>
      @type stdout
    </match>
    

    Environment

    • fluentd
    $ fluentd --version
    fluentd 1.12.3
    
    • OS
    $ lsb_release -d
    Description:    Ubuntu 18.04.6 LTS
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search