Question 109292 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Json – jq loop over objects and use shell command output to add a field

I’m trying to iterate over a JSON and add a field to each object in the JSON.
I’m doing this in a Gitlab pipeline stage but I don’t think is important.

The JSON is this:

{
 "service10": {
     "path_to_file" : "service10.txt"
     },
"service192": {
     "path_to_file" : "service192.txt"
     }
}

What I’m doing is using the path_to_file to check in a Gitlab repo if that file exists and I need to save the http response, the code now looks as follows:

check_files_existence:
  stage: update-external-repo
  script:
    - apt-get update --fix-missing
    - apt-get install jq -y
    - HTTP_RESPONSE_CODES=()

    # Check for file
    - |-
      for FILE_TO_CHECK in $(jq -r '.[].path_to_file ' $JSON_TO_READ); do
        
        last_status_code=($(curl [omitted]))
        
        HTTP_RESPONSE_CODES+=($last_status_code)      
        
       done

   # Write array of responses into artifact
    - echo "${HTTP_RESPONSE_CODES[@]}" >> ./http_response_code.txt

  artifacts:
    paths:
      - ./http_response_code.txt
    expire_in: 10 min

This generates a .txt file with the http responses separated by a whitespace.

What I want to do is add a field to each service while I loop over them with the http_response, so the JSON will be like:

{
 "service10": {
     "path_to_file" : "service10.txt",
     "http_response": 200
     },
"service192": {
     "path_to_file" : "service192.txt",
     "http_response": 404
     }
}

And then I’m going to export this an artifact to use it in another stage in my Gitlab Pipeline.

Using the above for loop I can get the file path but then it looks like I cannot update the JSON, if I use something like

for OBJECT in $(jq -r '.[] ' $JSON_TO_READ)

I cannot read the file path because using

jq '.path_to_file $OBJECT'

returns an error.

Is there a way to do this?

Thanks.

Tags:

2

Answers


  1. 0

    This probably calls for two separate copies of jq. Assuming your input file were changed to be valid JSON:

    jq -r 'to_entries[] | [.key, .value.path_to_file] | @tsv' <"$JSON_TO_READ" 
| while IFS=$'t' read -r key value; do
    # comment this next line in your real system
    (( ++http_status )) # stand-in so this can be tested without calling curl
    # uncomment this next line and replace the url in your real system
    #http_status=$(curl -s -o /dev/null -w "%{http_code}" "http://example.com/$key") || continue
    printf '%st%sn' "$key" "$http_status"
  done 
| jq -Rn --slurpfile input_file "$JSON_TO_READ" '
    reduce inputs as $inline
      ($input_file[0];
       (($inline | split("t")) as $pieces |
        .[$pieces[0]].http_response = ($pieces[1])))'

    Breaking down the shell pipeline:

    • First, we have a copy of jq that emits a tab-separated stream with your key and value. An entry here would look like:

      service10<TAB>service10.txt

    • Next, we have a shell pipeline (running a BashFAQ #1 loop) that reads these two variables, and runs curl (or in the example above, just adds an incrementing number as the status code) to determine the HTTP status code:

      service10<TAB>200

    • Finally, we have another copy of jq that accepts your input file via --slurpfile, and reads the stream of tab-separated service keys and HTTP status codes from stdin, using a reducer to incrementally update the status file as new values come in. (Because --slurpfile contains a list of JSON documents found in the file, not just a single document, we’re referring to $input_file[0] to refer to the first — and only — document in the file).

    Login or Signup to reply.
  2. 0

    If your file isn’t too big and you don’t mind reading the same input file multiple times, then here is another solution that should work:

    1. Count the number of key-value-pairs in the object
    2. Extract single entry by index from object
    3. Get path_to_file value from entry
    4. Call curl
    5. Add http_response to entry
    6. Use jq -s (--slurp) to reconstruct the object from its entries
    input='input.json'
length="$(jq 'length' "$input")"
for i in $(seq "$length"); do
  entry="$(jq --argjson idx "$i" 'to_entries[$idx-1]' "$input")"
  path="$(printf '%s' "$entry" | jq -r '.value.path_to_file')"
  last_status_code="$(curl ... "$path")"
  printf '%s' "$entry" | jq --argjson status "$last_status_code" '.value.http_response = $status'
done | jq -s 'from_entries'
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search