Question 110372 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Updating invalid json output to make it valid from a file

I have the following data that I already insert in a file called file.json:


{
  "ip": "1.1.1.1",
  "tested_count": 17,
  "last_scan_date": "1673169149"
}
{
  "ip": "1.1.1.1",
  "tested_count": 17,
  "last_scan_date": "1673169159"
}
{
  "ip": "1.1.1.1",
  "tested_count": 10,
  "last_scan_date": "1673169326"
}

To be specific, this is how I added it using jq every time I run the script.sh:

#!/bin/bash

# I run this script 3 times to insert into file.json:

#./script.sh 1.1.1.1 17
#./script.sh 1.1.1.1 17
#./script.sh 1.1.1.1 10

OUTPUT='file.json'
JSON_OUTPUT='data:{}'

IP="$1"
TEST_COUNT="$2"

JSON_OUTPUT=$(jq 
  --arg ip "${IP}" 
  --argjson tested_count "${TEST_COUNT}" 
  --arg last_scan_date "$(date +%s)" 
  '{ip: $ip, tested_count: $tested_count, last_scan_date: $last_scan_date}' <<<"${JSON_OUTPUT}")

# Write the output to the file
echo "${JSON_OUTPUT}" >>"${OUTPUT}"

Now, based on the output from file.json it looks like it is not a valid json. So, my aim is to transform that into a valid json that looks like below:

{
  "data": [
    {
      "ip": "1.1.1.1",
      "tested_count": 17,
      "last_scan_date": "1673169149"
    },
    {
      "ip": "1.1.1.1",
      "tested_count": 17,
      "last_scan_date": "1673169159"
    },
    {
      "ip": "1.1.1.1",
      "tested_count": 10,
      "last_scan_date": "1673169326"
    }
  ]
}

I have seen an example of this here but the example shown there is using a looping method which is not necessary in my case as my json data is dynamically added each time the script.sh above run.

What I have tried so far is using an update inputs filter like this but obviously, this does not fix the json as a valid syntax.

echo "$json_query" | jq -n '.data |= [inputs]' >> file.json
Tags:

2

Answers


  1. 0

    You can you sed:

    echo '{
  "ip": "1.1.1.1",
  "tested_count": 17,
  "last_scan_date": "1673169149"
}
{
  "ip": "1.1.1.1",
  "tested_count": 17,
  "last_scan_date": "1673169159"
}
{
  "ip": "1.1.1.1",
  "tested_count": 10,
  "last_scan_date": "1673169326"
}' | sed 's/}/},/;s/^/    /; 1 s/^/{n  "data": [n/; $ s/,/n  ]n}/'
    Login or Signup to reply.
  2. 0

    Doing it properly with jq.

    I added lots of comments for both the shell and the jq script parts.
    Don’t hesitate to question me in comments if something is unclear, anyway. That would be preferable to blindly copying code.

    #!/usr/bin/env bash

#./script.sh 1.1.1.1 17
#./script.sh 1.1.1.1 17
#./script.sh 1.1.1.1 10

# No reason to use ALL_CAPS variable names unless these are environment
output_file='file.json'

# ${var:?} expansion means var is mandatory and it will error if not provided
arg_ip="${1:?}"
arg_test_count="${2:?}"

# If output file does not exist
if ! [ -f "${output_file}" ]; then
  # Initializes it with object of empty JSON data array
  jq --null-input '{"data":[]}' > "${output_file}"
fi

# Buffer the output of processing updated JSON data
json_buffer=$(
  jq 
    --arg arg_ip "${arg_ip}" 
    --argjson tested_count "${arg_test_count}" 
    '
# Add new object entry to the data array
.data += [{
  "arg_ip": $arg_ip,
  "tested_count": $tested_count,
  "last_scan_date": now|rint # builtin jq timestamp integer part
}]
' "${output_file}"
)

# Write the buffer content back to the file
# printf '%sn' will preserve the buffer content
# and restores the trailing newline at end of file
printf '%sn' "${json_buffer}" > "${output_file}"
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search