skip to Main Content

I wish to process (call a parser on) the contents of a log event via fluend.
These come different formats (json, apache2, nginx and bespoke regexp), sometime I may wish to skip any processing.

Events come from the same source (i.e. kubernetes logs) so the I cannot tag by source.
Currently I’m using vanilla fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch.

There doesn’t appear to be a way of branching logic (there is only if there is no if else). So thus far I have only come across the following means of rewriting tag achieve this.

### Re-write the tag from log_format
<match **> 
  @type rewrite_tag_filter
  <rule>
    key kubernetes.container_name
    pattern /^(ingess-nginx)$/
    tag log.nginx
  </rule>
</match>

# Apache2 parser
<filter log.apache2>
  @type parser
  key_name log 
  reserve_data true
  inject_key_prefix log.
  remove_key_name_field true
  <parse>
    @type apache2
    keep_time_key true
  </parse>
</filter>

# Nginx parser
<filter log.nginx>
  @type parser
  key_name log 
  reserve_data true
  inject_key_prefix log.
  remove_key_name_field true
  <parse>
    @type nginx
    keep_time_key true
  </parse>
</filter>

<match **>
   @type elasticsearch
   @id out_es
   @log_level info
   include_tag_key true
...

This doesn’t work. Logs that were arriving at Elastic Search are no longer doing so.
The issue appears to be within the first clause that attempts to re-write the tag, although there are no errors and no indication as to what is going wrong. If I remove this paragraph event return to Elasticsearch, but obviously without the desire processing.

So how do I conditionally format (by way of the parser plugin) based on a field value?
Ideally I would prefer not to use container_name but an annotation which may not be present, is there a way of making the rewrite only occur is a particular field exists? and otherwise process as normal.

2

Answers


  1. Chosen as BEST ANSWER

    Attributes from kubernetes are nested (there is a paucity of documentation as to why) but consequently this requires a different notation. Note the key line.

    ### Re-write the tag from log_format
    <match **> 
      @type rewrite_tag_filter
      <rule>
        key $.kubernetes.container_name
        pattern /^(ingess-nginx)$/
        tag log.nginx
      </rule>
    </match>
    

  2. Here’s a discussion with one possible solution to this use case. In this case it was important to implement a catch-all to allow messages to be passed on to the output. fujimotos also mentions an out_copy approach.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search