Question 85226 in Docker Question posted in
The official documentation can be found here.

JSON with jq into bash array / loop through for proper alerting output – Docker

I have a JSON result which I want to go through and send an alert, if one or several of the values in the JSON string is smashing thresholds.

This bash command:

for sat in `docker exec -i storagenode wget -qO - localhost:14002/api/sno | jq .satellites[].id -r`; do docker exec -i storagenode wget -qO - localhost:14002/api/sno/satellite/$sat | jq .id,.audits; done

Provides the following (excerpt):

"12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"
{
  "auditScore": 1,
  "suspensionScore": 1,
  "onlineScore": 0.9974358974358974,
  "satelliteName": "us2.storj.io:7777"
}
"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"
{
  "auditScore": 1,
  "suspensionScore": 1,
  "onlineScore": 0.9989041005632043,
  "satelliteName": "saltlake.tardigrade.io:7777"
}

Now I want to go through the results and e.g. in case, onlineScore drops below 0.9 (or suspensionScore resp. auditScore below 1.0, I want to create a text alert, which also includes the satelliteName. Example: auditScore below threshold: 0.98 for us2.storj.io

I guess I can start with the following (from https://unix.stackexchange.com/questions/413878/json-array-to-bash-variables-using-jq), but I do not know, how to loop through the results and how to name and validate the fields:

jq -r '.[] | to_entries | .[] | .key + "=" + (.value | @sh)'

What the above mentioned statement delivers in subtleties:

docker exec -i storagenode wget -qO - localhost:14002/api/sno

provides:

{"nodeID":"1veqEG5xuBNkt...","wallet":"12345","walletFeatures":["zksync"],"satellites":[{"id":"12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo","url":"us2.storj.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":4592556672},{"id":"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE","url":"saltlake.tardigrade.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":513269323264},{"id":"121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6","url":"ap1.storj.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":70956116864},{"id":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","url":"us1.storj.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":322340591104},{"id":"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs","url":"eu1.storj.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":148740125312},{"id":"12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB","url":"europe-north-1.tardigrade.io:7777","disqualified":null,"suspended":null,"currentStorageUsed":100681406976}],"diskSpace":{"used":1162257094528,"available":9500000000000,"trash":43196690332,"overused":0},"bandwidth":{"used":31670206976,"available":0},"lastPinged":"2022-01-02T07:55:51.886776586Z","version":"1.45.3","allowedVersion":"1.24.0","upToDate":true,"startedAt":"2021-12-31T00:00:32.209840775Z"}

And this one:

docker exec -i storagenode wget -qO - localhost:14002/api/sno/satellite/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE

provides:

{"id":"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE","storageDaily":[{"atRestTotal":11726579066524.266,"intervalStart":"2022-01-01T00:00:00Z"}],"bandwidthDaily":[{"egress":{"repair":390576640,"audit":7424,"usage":2373615872},"ingress":{"repair":2745100032,"usage":26744320},"delete":0,"intervalStart":"2022-01-01T00:00:00Z"},{"egress":{"repair":143168256,"audit":3584,"usage":786989568},"ingress":{"repair":1034401280,"usage":8107264},"delete":0,"intervalStart":"2022-01-02T00:00:00Z"}],"storageSummary":11726579066524.266,"bandwidthSummary":7508714240,"egressSummary":3694361344,"ingressSummary":3814352896,"currentStorageUsed":513269323264,"audits":{"auditScore":1,"suspensionScore":1,"onlineScore":0.9989041005632043,"satelliteName":"saltlake.tardigrade.io:7777"},"auditHistory":{"score":0.9989041005632043,"windows":[{"windowStart":"2021-12-03T00:00:00Z","totalCount":36,"onlineCount":36},{"windowStart":"2021-12-03T12:00:00Z","totalCount":30,"onlineCount":30},{"windowStart":"2021-12-04T00:00:00Z","totalCount":43,"onlineCount":42},{"windowStart":"2021-12-04T12:00:00Z","totalCount":68,"onlineCount":68},{"windowStart":"2021-12-05T00:00:00Z","totalCount":53,"onlineCount":53},{"windowStart":"2021-12-05T12:00:00Z","totalCount":36,"onlineCount":36},{"windowStart":"2021-12-06T00:00:00Z","totalCount":33,"onlineCount":33},{"windowStart":"2021-12-06T12:00:00Z","totalCount":53,"onlineCount":53},{"windowStart":"2021-12-07T00:00:00Z","totalCount":27,"onlineCount":27},{"windowStart":"2021-12-07T12:00:00Z","totalCount":65,"onlineCount":65},{"windowStart":"2021-12-08T00:00:00Z","totalCount":30,"onlineCount":30},{"windowStart":"2021-12-08T12:00:00Z","totalCount":27,"onlineCount":27},{"windowStart":"2021-12-09T00:00:00Z","totalCount":46,"onlineCount":46},{"windowStart":"2021-12-09T12:00:00Z","totalCount":48,"onlineCount":48},{"windowStart":"2021-12-10T00:00:00Z","totalCount":49,"onlineCount":49},{"windowStart":"2021-12-10T12:00:00Z","totalCount":67,"onlineCount":67},{"windowStart":"2021-12-11T00:00:00Z","totalCount":71,"onlineCount":71},{"windowStart":"2021-12-11T12:00:00Z","totalCount":52,"onlineCount":52},{"windowStart":"2021-12-12T00:00:00Z","totalCount":59,"onlineCount":59},{"windowStart":"2021-12-12T12:00:00Z","totalCount":77,"onlineCount":77},{"windowStart":"2021-12-13T00:00:00Z","totalCount":79,"onlineCount":79},{"windowStart":"2021-12-13T12:00:00Z","totalCount":79,"onlineCount":79},{"windowStart":"2021-12-14T00:00:00Z","totalCount":65,"onlineCount":65},{"windowStart":"2021-12-14T12:00:00Z","totalCount":59,"onlineCount":59},{"windowStart":"2021-12-15T00:00:00Z","totalCount":87,"onlineCount":87},{"windowStart":"2021-12-15T12:00:00Z","totalCount":82,"onlineCount":81},{"windowStart":"2021-12-16T00:00:00Z","totalCount":96,"onlineCount":96},{"windowStart":"2021-12-16T12:00:00Z","totalCount":66,"onlineCount":64},{"windowStart":"2021-12-17T00:00:00Z","totalCount":36,"onlineCount":36},{"windowStart":"2021-12-17T12:00:00Z","totalCount":48,"onlineCount":48},{"windowStart":"2021-12-18T00:00:00Z","totalCount":37,"onlineCount":37},{"windowStart":"2021-12-18T12:00:00Z","totalCount":60,"onlineCount":60},{"windowStart":"2021-12-19T00:00:00Z","totalCount":69,"onlineCount":69},{"windowStart":"2021-12-19T12:00:00Z","totalCount":32,"onlineCount":32},{"windowStart":"2021-12-20T00:00:00Z","totalCount":53,"onlineCount":53},{"windowStart":"2021-12-20T12:00:00Z","totalCount":37,"onlineCount":37},{"windowStart":"2021-12-21T00:00:00Z","totalCount":80,"onlineCount":80},{"windowStart":"2021-12-21T12:00:00Z","totalCount":57,"onlineCount":57},{"windowStart":"2021-12-22T00:00:00Z","totalCount":46,"onlineCount":46},{"windowStart":"2021-12-22T12:00:00Z","totalCount":33,"onlineCount":33},{"windowStart":"2021-12-23T00:00:00Z","totalCount":42,"onlineCount":42},{"windowStart":"2021-12-23T12:00:00Z","totalCount":73,"onlineCount":73},{"windowStart":"2021-12-24T00:00:00Z","totalCount":35,"onlineCount":35},{"windowStart":"2021-12-24T12:00:00Z","totalCount":44,"onlineCount":44},{"windowStart":"2021-12-25T00:00:00Z","totalCount":81,"onlineCount":81},{"windowStart":"2021-12-25T12:00:00Z","totalCount":43,"onlineCount":43},{"windowStart":"2021-12-26T00:00:00Z","totalCount":62,"onlineCount":62},{"windowStart":"2021-12-26T12:00:00Z","totalCount":79,"onlineCount":79},{"windowStart":"2021-12-27T00:00:00Z","totalCount":70,"onlineCount":70},{"windowStart":"2021-12-27T12:00:00Z","totalCount":90,"onlineCount":90},{"windowStart":"2021-12-28T00:00:00Z","totalCount":65,"onlineCount":65},{"windowStart":"2021-12-28T12:00:00Z","totalCount":77,"onlineCount":77},{"windowStart":"2021-12-29T00:00:00Z","totalCount":83,"onlineCount":83},{"windowStart":"2021-12-29T12:00:00Z","totalCount":99,"onlineCount":99},{"windowStart":"2021-12-30T00:00:00Z","totalCount":74,"onlineCount":74},{"windowStart":"2021-12-30T12:00:00Z","totalCount":84,"onlineCount":84},{"windowStart":"2021-12-31T00:00:00Z","totalCount":70,"onlineCount":70},{"windowStart":"2021-12-31T12:00:00Z","totalCount":93,"onlineCount":93},{"windowStart":"2022-01-01T00:00:00Z","totalCount":120,"onlineCount":120},{"windowStart":"2022-01-01T12:00:00Z","totalCount":112,"onlineCount":112},{"windowStart":"2022-01-02T00:00:00Z","totalCount":46,"onlineCount":46}]},"priceModel":{"EgressBandwidth":2000,"RepairBandwidth":1000,"AuditBandwidth":1000,"DiskSpace":150},"nodeJoinedAt":"2021-05-11T20:11:14.910165Z"}

Sorry, not sure how to better format the huge json content here in a better way.

Thank you for your help. I am not sure how I can modify the output, loop through the results and create the corresponding target output. jq as a tool is new for me and the documentation is not yet self-explanatory for me.

UPDATE #1

There is a result for http://localhost:14002/api/sno/satellites, I was not aware of before. The result is:

{
...
   "storageSummary": 6.8624392E13,
...
   "audits": [
      {
         "auditScore": 1,
         "suspensionScore": 1,
         "onlineScore": 0.99743587,
         "satelliteName": "us2.storj.io:7777"
      },
      {
         "auditScore": 1,
         "suspensionScore": 1,
         "onlineScore": 0.9992917,
         "satelliteName": "saltlake.tardigrade.io:7777"
      },
...
      {
         "auditScore": 1,
         "suspensionScore": 1,
         "onlineScore": 0.99930555,
         "satelliteName": "ap1.storj.io:7777"
      }
   ]
}

I think that makes the required handling much easier at the end, right?

Tags:

2

Answers


  1. 0

    Add threshold variables and values using --argjson (preferred over --arg as we talk about numbers), then extract from the .audits object each item that smashes the corresponding threshold using select, format them in a way to carry the actual value and the threshold to be reused in the output, and finally convert those items into an alert string (which will be output as is due to the --raw-output parameter).

    for sat in `docker exec -i storagenode wget -qO - localhost:14002/api/sno | jq .satellites[].id -r`
do
  docker exec -i storagenode wget -qO - localhost:14002/api/sno/satellite/$sat 
  | jq --raw-output 
    --argjson auditThreshold 1 
    --argjson suspensionThreshold 1 
    --argjson onlineThreshold 1 
    '.audits
      | .satelliteName as $name
      | (
          [{auditScore}, $auditThreshold],
          [{suspensionScore}, $suspensionThreshold],
          [{onlineScore}, $onlineThreshold]
        )
      | select(.[0][] < .[1])
      | "(.[0] | keys[]) ((.[0][])) below threshold ((.[1])) for ($name)"
    '
done

    onlineScore (0.9989041005632043) below threshold (1) for saltlake.tardigrade.io:7777

    Simplified Demo

    Login or Signup to reply.
  2. 0

    You can produce an output which can readable to a while read loop (i.e. while read audit_score suspension_score online_score satellite_name; do) by changing jq .id,.audits to jq -r '.audits | "(.auditScore) (.suspensionScore) (.onlineScore) (.satelliteName)"'. Use bc as a comparison tool since Bash doesn’t support floating points.

    Also this took me a while to compose but you can compose the messages directly using this:

    jq -r '.audits | 
    (select(.auditScore < 1) | "auditScore below threshold: (.auditScore) for (.satelliteName)"),
    (select(.suspensionScore < 1) | "suspensionScore below threshold: (.suspensionScore) for (.satelliteName)"),
    (select(.onlineScore < 1) | "onlineScore below threshold: (.onlineScore) for (.satelliteName)")
'

    It should work even with merged outputs from this:

    docker exec -i storagenode wget -qO - localhost:14002/api/sno | jq .satellites[].id -r | 
    xargs -I'{}' docker exec -i storagenode wget -qO - 'localhost:14002/api/sno/satellite/{}'
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search