I wish to process (call a parser on) the contents of a log event via fluend.
These come different formats (json, apache2, nginx and bespoke regexp), sometime I may wish to skip any processing.
Events come from the same source (i.e. kubernetes logs) so the I cannot tag by source.
Currently I’m using vanilla fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch.
There doesn’t appear to be a way of branching logic (there is only if there is no if else). So thus far I have only come across the following means of rewriting tag achieve this.
### Re-write the tag from log_format
<match **>
@type rewrite_tag_filter
<rule>
key kubernetes.container_name
pattern /^(ingess-nginx)$/
tag log.nginx
</rule>
</match>
# Apache2 parser
<filter log.apache2>
@type parser
key_name log
reserve_data true
inject_key_prefix log.
remove_key_name_field true
<parse>
@type apache2
keep_time_key true
</parse>
</filter>
# Nginx parser
<filter log.nginx>
@type parser
key_name log
reserve_data true
inject_key_prefix log.
remove_key_name_field true
<parse>
@type nginx
keep_time_key true
</parse>
</filter>
<match **>
@type elasticsearch
@id out_es
@log_level info
include_tag_key true
...
This doesn’t work. Logs that were arriving at Elastic Search are no longer doing so.
The issue appears to be within the first clause that attempts to re-write the tag, although there are no errors and no indication as to what is going wrong. If I remove this paragraph event return to Elasticsearch, but obviously without the desire processing.
So how do I conditionally format (by way of the parser plugin) based on a field value?
Ideally I would prefer not to use container_name but an annotation which may not be present, is there a way of making the rewrite only occur is a particular field exists? and otherwise process as normal.
2
Answers
Attributes from kubernetes are nested (there is a paucity of documentation as to why) but consequently this requires a different notation. Note the
key
line.Here’s a discussion with one possible solution to this use case. In this case it was important to implement a catch-all to allow messages to be passed on to the output. fujimotos also mentions an
out_copy
approach.