Question 109904 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Bash: Ignore key value pairs from a JSON that failed to parse using jq

I’m writing a bash script to read a JSON file and export the key-value pairs as environment variables. Though I could extract the key-value pairs, I’m struggling to skip those entries that failed to parse by jq.

JSON (key3 should fail to parse)

{
 "KEY1":"ABC",
 "KEY2":"XYZ",
 "KEY3":"---ABC---n
dskfjlksfj"

}

Here is what I tried

for pair in $(cat test.json  | jq -r -R  '. as $line | try fromjson catch $line | to_entries | map("(.key)=(.value)") | .[]' ); do
    echo $pair
    export $pair
done

And this is the error

jq: error (at <stdin>:1): string ("{") has no keys
jq: error (at <stdin>:2): string ("  "key1...) has no keys

My code is based on these posts:

  1. How to convert a JSON object to key=value format in jq?
  2. How to ignore broken JSON line in jq?
  3. Ignore Unparseable JSON with jq
Tags:

4

Answers


  1. 0

    [Note: this response was made to the original question, which has since been changed. The response essentially assumes the input consists of JSONLines interspersed with other lines.)

    Since the goal seems to be to ignore lines that don’t have valid key-value pairs, you can simply use catch empty:

    while read -r line ; do 
    echo export "$line"
done < <(test.json jq -r -R  '
  try fromjson catch empty
  | objects
  | to_entries[]
  | "(.key)="(.value|@sh)"" 
')

    Note also the use of @sh and of the shell’s read, and the fact that .value (in jq) and $line (in the shell) are both quoted. These are all important for robustness, though further refinements might still be necessary for additional robustness.

    Login or Signup to reply.
  2. 0

    Here’s a response to the revised question. Unfortunately, it will only be useful in certain limited cases, not including the example you give. (Basically, it depends on jq’s parser being able to recover before the end of file.)

    while read -r line ; do 
    echo export "$line"
done < <(< test.json jq -rn '
  def do:
    try inputs catch null
    | objects
    | to_entries[]
    | "(.key)="(.value|@sh)"" ;
  recurse(do) | select(.)
')

    Note that further refinements may be warranted, especially if there is potentially something fishy about the key names being used as shell variable names.

    Login or Signup to reply.
  3. 0

    Perhaps there is an algorithm that will repair the broken JSON produced by the upstream system. If not, the following is a horrible but possibly useful "hack" that will at least capture KEY1 and KEY2 in the example in the Q:

    jq -Rr '
   capture(""(?<key>[^"]*)"[ t]*:[ t]*(?<value>[^}]+)") 
   | (.value |= sub("[ t]+$"; "") )  # trailing whitespace
   | if .value|test("^".*"") then .value |= sub(""[ t]*[,}[ t]*$"; """) else . end
   | select(.value | test("^".*"$") or (contains(""")|not) )  # a string or not a string
   | "(.key)=(.value|@sh)" 
'
    Login or Signup to reply.
  4. 0

    The broken JSON in the example could be repaired in a number of ways, e.g.:

    sed '/\n$/{N; s/\nn/\n/;}'

    produces:

    {
 "KEY1":"ABC",
 "KEY2":"XYZ",
 "KEY3":"---ABC---ndskfjlksfj"

}

    At least that’s JSON 🙂

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search