I have this fluentd configuration:
<source>
@type tail
<parse>
@type regexp
expression /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>w+) (?<path>[^ ]*) (?<http>[^ ]*)" (?<status_code>[^ ]*) (?<size>[^ ]*)(?:s"(?<referer>[^"]*)") "(?<agent>[^"]*)" (?<urt>[^"]*).*/
time_format %d/%b/%Y:%H:%M:%S %z
keep_time_key true
types size:integer,reqtime:float,uct:float,uht:float,urt:float
</parse>
path /var/log/nginx/access.log
pos_file /tmp/fluent_nginx.pos
tag nginx
</source>
My log format:
193.137.78.17 - - [07/Jan/2023:09:21:59 +0000] "GET /net/api/employee HTTP/1.1" 200 2323 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 0.014
193.137.78.17 - - [07/Jan/2023:09:22:00 +0000] "GET /net/api/employee HTTP/1.1" 200 2323 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 0.005
I’ve tested my regex on regex101 and works without problems. Still, I get a no patterns matched warning on fluentd. I don’t understand why the log isn’t parsed correctly.
Jan 07 09:26:26 srv-api fluentd[14878]: 2023-01-07 09:26:26 +0000 [warn]: #0 no patterns matched tag="nginx"
Can anyone help me, please? Thanks!
2
Answers
I think your problem is leading spaces in the log
Your pattern is insisting that the
<remote>has no spaces before it, but you do have 4 spaces in your log before the remote IP.The simplest way, to my mind, is to insert an optional variable-number-of-spaces at the beginning.
How it works
The
(and)are just to make life easier for people reading the code: they will see that between them is a space character, which they might not otherwise notice.The
*means 0 or more of these.This allows 0 or more spaces at the beginning of the line to be matched and discarded.
Incidentally
I noticed you are sometimes escaping
"withand sometimes not. Is there a reason for this?You should directly use the nginx parser plugin instead.
Here is a complete working example with the sample input plugin and the nginx parser plugin:
fluent-nginx-test.conf
Run
Output
Apart from that, I used your regex with the regexp parser plugin and it’s working fine too (though there are redundant values in
typesfield):fluent-nginx-test-with-regexp.conf
Run
Output
However, the error
no patterns matched tag="nginx"in message:which means that you don’t have a corresponding
matchsection in your config file. You must have amatchsection with the respectivetagthat you want to process or output.Example:
You should take a look at the config file syntax specifically How do the match patterns work? section for more guidelines.
Also, you might want to use the vscode-fluentd extension for syntax highlighting with VS Code.
Environment
fluentd