Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Json – Logstash replace colon with arrow

My goal is to read data from kinesis and upload csv file on s3, but in csv file arrow(=>) is coming in json data insted of colon(:)

Logstash config file

input {
  kinesis {
    application_name => "logstash"
    kinesis_stream_name => "events"
    type => "kinesis"
    region => "us-east-1"
    profile => "default"
    metrics => "cloudwatch"
    codec => "json"
  }
}

filter {
    grok {
      match => { "[data][ti]" => "%{YEAR:event_year}-%{MONTHNUM2:event_month}-%{MONTHDAY:event_day}" }
   }
}

output {
  s3 {
    bucket => "test-logstash"
    region => "us-east-1"
    prefix => "%{event_year}/%{event_month}/%{event_day}/%{[data][brandid]}/%{[data][event_name]}"
    encoding => "none"
    codec => csv {
        separator => "␁"
    }
    size_file => 200000000
    time_file => 60
  }
}
Tags:

2

Answers


  1. 0

    When writing your Hash to the CSV file, convert it to JSON using as_json or a JSON string using to_json.

    Login or Signup to reply.
  2. 0

    You can use the mutate filter in Logstash to replace "arrow (=>)" with "colon (:)".

    input {
  kinesis {
    application_name => "logstash"
    kinesis_stream_name => "events"
    type => "kinesis"
    region => "us-east-1"
    profile => "default"
    metrics => "cloudwatch"
    codec => "json"
  }
}

filter {
  # Use the mutate filter to replace "arrow (=>)" with "colon (:)"
  mutate {
    gsub => [
      "message", "=>", ":"
    ]
  }

  grok {
    match => { "[data][ti]" => "%{YEAR:event_year}-%{MONTHNUM2:event_month}-%{MONTHDAY:event_day}" }
  }
}

output {
  s3 {
    bucket => "test-logstash"
    region => "us-east-1"
    prefix => "%{event_year}/%{event_month}/%{event_day}/%{[data][brandid]}/%{[data][event_name]}"
    encoding => "none"
    codec => csv {
      separator => "␁"
    }
    size_file => 200000000
    time_file => 60
  }
}

    In the filter section, the mutate filter is used to perform a global substitution (gsub) to replace "arrow (=>)" with "colon (:)". This should ensure that your JSON data is correctly processed as expected.

    Make sure to adjust the field name (message) in the mutate filter if your JSON data is located in a different field.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search