Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Сreating multilevel json with the jq utility in a bash script

I get the contents of a text file into an array and then sequentially get the elements of the array and try to add the elements to the json:

#!/bin/bash

JSON=$(jq -n '')
readarray -t array < ./station.sample
count=0

for e in "${array[@]}"
do
    if echo "$e" | grep -Eq '^[0-9]{10}' >/dev/null
      then
        timestamp=$e
        datetime=`date +'%Y-%m-%d %H:%M:%S' -d "@$e"`
        JSON=$(echo $JSON | jq --arg timestamp "${timestamp}" '. += $ARGS.named')
        JSON=$(echo $JSON | jq --arg datetime "${datetime}" '. += $ARGS.named')
    fi

    if echo "$e" | grep '^Station ' >/dev/null
      then
        NODE=$(jq -n '')
        mac=`echo "$e" | awk '{ print $2 }'`
        interface=`echo "$e" | awk '{ print $4 }' | rev | cut -c2- | rev`

        JSON=$(echo $JSON | jq --argjson nodes "[]" '. += $ARGS.named')
        JSON=$(echo $JSON | jq --arg mac "${mac}" --arg count "${count}" '.nodes[0] += {"mac": $mac}')
    JSON=$(echo $JSON | jq --arg interface "${interface}" '.nodes[0] += {"interface": $interface}')

        count=$((count+1))
    fi
done

As a result, I get a json like this:

{
  "timestamp": "1721396365",
  "datetime": "2024-07-19 16:39:25",
  "nodes": [
    {
      "mac": "14:88:00:00:00:06"
    }
  ]
}

But I need to add multiple objects to the list, so I set a count variable and want to use it to specify the index of the list:

JSON=$(echo $JSON | jq --arg mac "${mac}" --arg count "${count}" '.nodes[$count] += {"mac": $mac}')

But in this variant adding list items doesn’t work.

How can I dynamically specify the index of the list?

I’ve tried all the options that came to mind, but I haven’t found a solution yet:

 NODE=$(echo $NODE | jq --arg mac "$mac" '. += $ARGS.named')
 NODE=$(echo $NODE | jq --arg interface "${interface}" '. += $ARGS.named')
 JSON=$(echo $JSON | jq --argjson node "${NODE}" '.nodes[.nodes | length] = $ARGS.named')

It is necessary to use a variable to specify the index of the list.

Tags:

2

Answers


  1. 0

    .nodes[$count] += {"mac": $mac}

    But in this variant adding list items doesn’t work.

    That’s because when you use += for arrays, the RHS must be an array, like so:

     .nodes[$count] += [{"mac": $mac}]

    By the way, I suspect you could greatly simplify things by using jq to do all (or at least nearly all) the required manipulations, thus avoiding the bash/awk/grep messiness.

    Login or Signup to reply.
  2. 0

    It was interesting to deep dive into jq scripting, and here it is.

    There is no clue as to what the input is and what the desired output is, so here are my assumptions:

    • Input data is lines starting with Unix timestamps or starting with Station (nodes), and other ignored lines.
      Simple input generator:

      for _i in {0..10}; do
  case $_i in
  1 | 7 | 8)
    echo "$(date +%s -d "now - $((10 - $_i)) days") rest..";;
  4)
    echo "other log line";;
  *)
    printf "Station 14:88:00:00:00:%02X some   iface%d.12x rest...n" "$_i" "$_i";;
  esac;
done >./station.sample
    • Output is a grouping of stations/nodes ("mac" and "iface") by unix timestamps.
      The result for test input above:

      [
  {
    "nodes": [
      {"mac": "14:88:00:00:00:00", "interface": "iface0.12"}
    ]
  }, {
    "timestamp": 1721465934, "datetime": "2024-07-20 08:58:54",
    "nodes": [
      {"mac": "14:88:00:00:00:02", "interface": "iface2.12"},
      {"mac": "14:88:00:00:00:03", "interface": "iface3.12"},
      {"mac": "14:88:00:00:00:05", "interface": "iface5.12"},
      {"mac": "14:88:00:00:00:06", "interface": "iface6.12"}
    ]
  }, {
    "timestamp": 1721984334, "datetime": "2024-07-26 08:58:54",
    "nodes": []
  }, {
    "timestamp": 1722070734, "datetime": "2024-07-27 08:58:54",
    "nodes": [
      {"mac": "14:88:00:00:00:09", "interface": "iface9.12"},
      {"mac": "14:88:00:00:00:0A", "interface": "iface10.12"}
    ]
  }
]

    Here is jq filter, save this into filter.jq:

    [
  inputs |                     # read input file line by line without parsing, this requere -Rn flags
  if test("^\d{10}") then     # parse line with unix timestamps
    .[0:10] | tonumber | {
      timestamp: .,
      datetime: gmtime | strftime("%Y-%m-%d %H:%M:%S"),
    }
  elif test("^Station ") then  # parse line with "station"
    [splits("\s+")] | {
      mac: .[1],
      interface: .[3][:-1],    # trim last character
    }
  else                         # other lines, skip them
    empty
  end
] + [{timestamp: null}] |      # append empty object for complete last group in the following 'foreach'
[
  foreach .[] as $_o (         # groupping timestamps and stations/nodes
    [[], []];
    if $_o | has("timestamp") then [[$_o], .[0]] else [(.[0] + [$_o]), []] end;
    if ($_o | has("timestamp")) and (.[1] | length) > 0 then
      if .[1][0] | has("timestamp") then
        .[1][0] + {
          nodes: .[1][1:]
        }
      else
        {nodes: .[1]}          # case without Unix timestamp (file beginning)
      end
    else
      empty
    end
  )
]

    This filter should be run with the -Rn flags, e.g.:

    jq -Rn -f ./filter.jq <./station.sample
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search