Question 83921 in Docker Question posted in
The official documentation can be found here.

Fluentd – Kubernetes – ParserError error="pattern not matched with data" – Docker

I’m trying to redirect Kubernetes logs from containers to OpenSearch.
But there is always some error with the date. What am I doing wrong?

Docker logs example:

{"log":"time="2022-04-01T10:02:31Z" level=warning msg="Cannot take snapshot backup" controller=longhorn-backup error="could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'" node=k8s-worker2n","stream":"stderr","time":"2022-04-01T10:02:31.230191143Z"}
{"log":"E0401 10:02:31.230146       1 backup_controller.go:153] longhorn-backup: fail to sync backup longhorn-system/backup-989764daba094e0d: could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'n","stream":"stderr","time":"2022-04-01T10:02:31.230214608Z"}
{"log":"time="2022-04-01T10:02:31Z" level=warning msg="Dropping Longhorn backup longhorn-system/backup-989764daba094e0d out of the queue" controller=longhorn-backup error="longhorn-backup: fail to sync backup longhorn-system/backup-989764daba094e0d: could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'" node=k8s-worker2n","stream":"stderr","time":"2022-04-01T10:02:31.230218285Z"}

Fluentd Output:

fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time="2022-04-01T09:54:33Z" level=warning msg="Cannot take snapshot backup" controller=longhorn-backup error="could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'" node=k8s-worker1n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.524389862 +0000 record={"log"=>"time="2022-04-01T09:54:33Z" level=warning msg="Cannot take snapshot backup" controller=longhorn-backup error="could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'" node=k8s-worker1n", "stream"=>"stderr"}
fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time="2022-04-01T09:54:33Z" level=warning msg="Error syncing Longhorn backup longhorn-system/backup-c5104cf80da04be6" controller=longhorn-backup error="longhorn-backup: fail to sync backup longhorn-system/backup-c5104cf80da04be6: could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'" node=k8s-worker1n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.524404952 +0000 record={"log"=>"time="2022-04-01T09:54:33Z" level=warning msg="Error syncing Longhorn backup longhorn-system/backup-c5104cf80da04be6" controller=longhorn-backup error="longhorn-backup: fail to sync backup longhorn-system/backup-c5104cf80da04be6: could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'" node=k8s-worker1n", "stream"=>"stderr"}
fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time="2022-04-01T09:54:33Z" level=warning msg="Cannot take snapshot backup" controller=longhorn-backup error="could not find snapshot 'snapshot-9d5705bf-26fc-49e5-a771-cf9352049c04' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'" node=k8s-worker1n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.538539106 +0000 record={"log"=>"time="2022-04-01T09:54:33Z" level=warning msg="Cannot take snapshot backup" controller=longhorn-backup error="could not find snapshot 'snapshot-9d5705bf-26fc-49e5-a771-cf9352049c04' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'" node=k8s-worker1n", "stream"=>"stderr"}

Config:

<source>
  @type tail
  @id tail_all_container_logs
  @label @FLUENTD.OPENSEARCH
  path /var/log/containers/longhorn*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  exclude_path "#{ENV['FLUENT_ALL_CONTAINERS_TAIL_EXCLUDE_PATHS']}"
  <parse>
      @type json
  </parse>
</source>

<filter kubernetes.**>
    @type parser
    key_name log
    <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%N%z
        timezone +00:00
    </parse>
</filter>
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    @piotr-malec - Thanks for the answer. I've tried many different date/time formats but it's always the same.

    <filter kubernetes.**>
    @type parser
    key_name log
    <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%N%z
        #time_format %Y-%m-%dT%H:%M:%S.%z
        #time_format %Y-%m-%dT%H:%M:%S%z
        #time_format %Y-%m-%dT%H:%M:%Sz
        #time_format %Y-%m-%dT%H:%M:%SZ
        #time_format %Y-%m-%dT%H:%M:%S%Z
        #time_format %Y-%m-%dT%H:%M:%S.%z
        timezone +00:00
    </parse>
</filter>

    Now I have a similar case with a different log.

    fluentd/fluentd-9gfjs[fluentd]: 2022-04-05 08:46:58 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '[2022-04-05T08:46:58,305][INFO ][o.o.j.s.JobSweeper       ] [opensearch-cluster-master-0] Running full sweepn'" location=nil tag="opensearch-master" time=1970-01-01 00:33:42.306044198 +0000 record={"log"=>"[2022-04-05T08:46:58,305][INFO ][o.o.j.s.JobSweeper       ] [opensearch-cluster-master-0] Running full sweepn", "stream"=>"stdout"}
fluentd/fluentd-9gfjs[fluentd]: 2022-04-05 08:51:58 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '[2022-04-05T08:51:58,306][INFO ][o.o.j.s.JobSweeper       ] [opensearch-cluster-master-0] Running full sweepn'" location=nil tag="opensearch-master" time=1970-01-01 00:33:42.306370871 +0000 record={"log"=>"[2022-04-05T08:51:58,306][INFO ][o.o.j.s.JobSweeper       ] [opensearch-cluster-master-0] Running full sweepn", "stream"=>"stdout"}

    I have tried the following formats. Each of them does not work.

    <filter opensearch-master>
    @type parser
    key_name log
    <parse>
        @type json
        time_key @timestamp
        time_format %Y-%m-%dT%H:%M:%S,%3N
        #time_format %Y-%m-%dT%H:%M:%S,%L
        timezone +02:00
    </parse>
</filter>

    Ruby Documentation: https://docs.ruby-lang.org/en/2.4.0/Time.html#method-i-strftime


  2. 0

    The date/time from your logs is 2022-04-01T09:54:33Z note no milliseconds.

    While in your config has time_format %Y-%m-%dT%H:%M:%S.%N%z

    %N – Fractional seconds digits, default is 9 digits (nanosecond).

    Try to remove the .%N part to match with your time format from logs. Which would be:

    time_format %Y-%m-%dT%H:%M:%S%z

    For more information about the time format syntax kindly refer to this page.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search