Amazon web services – AWS query string does not extact any data when executed by .sh script

I’m new to Bash scripting and would like to test extracting a data, my plan is to extract data to a .txt file base on the aws query from .sh file (bashscript). I have a bigger purpose but I want to have this work first before i input my main aws query.

I have a .sh script here below.

NOTE: I put all values to -confidential- without any quotations. Just the value

#!/bin/bash

variables () {
lambdas="confidential"
}
variables

queryId=$(aws logs start-query --log-group-name $lambdas --start-time 1695704400 --end-time 1695707999 --query-string 'filter @message like "requestId" | parse @message "operationName: '*'," as API | parse @message "status: *," as Status | parse @message "channel: '*'," as Channel | stats count(*) as Total by API,Channel,Status | sort API,Total desc' | jq '.queryId' | awk -F '"' '{print $2}')


results=$(aws logs get-query-results --query-id $queryId | jq .)


echo "$results" > testextract5.txt

Whenever I run that .sh file, this is the result .txt below:

{
  "results": [],
  "statistics": {
    "recordsMatched": 0,
    "recordsScanned": 0,
    "bytesScanned": 0
  },
  "status": "Running"
}

BUT when I manually copy paste each command from .sh file all the bash script in the git-bash command line, it works! See below:

jomasangkay@DESKTOP-R1DM530:/mnt/c/Users/personal pc/Desktop/company/Bashscript/test$ queryId=$(aws logs start-query --log-group-name $lambdas --start-time 1695704400 --end-time 1695707999 --query-string 'filter @message like "requestId" | parse @message "operationName: '*'," as API | parse @message "status: *," as Status | parse @message "channel: '*'" as Channel | stats count(*) as Total by API,Channel,Status | sort API,Total desc' | jq '.queryId' | awk -F '"' '{print $2}')
jomasangkay@DESKTOP-R1DM530:/mnt/c/Users/personal pc/Desktop/company/Bashscript/test$ results=$(aws logs get-query-results --query-id $queryId | jq .)
jomasangkay@DESKTOP-R1DM530:/mnt/c/Users/personal pc/Desktop/company/Bashscript/test$ echo "$results" > testextract5.txt

RESULT extracted .txt below,

     {
        "field": "API",
        "value": "'Confidential'"
      },
      {
        "field": "Channel",
        "value": "'Confidential'"
      },
      {
        "field": "Status",
        "value": "Confidential"
      },
      {
        "field": "Total",
        "value": "1"
      }
    ]
  ],
  "statistics": {
    "recordsMatched": 16507,
    "recordsScanned": 69000,
    "bytesScanned": 29835564
  },
  "status": "Complete"
}

Action taken/Future Plan:

I tried to troubleshoot and used a bash -x script.sh but I don’t see any issue at all. I tried to rollback to wsl version 1, installed latest update of aws cli. None of it works. Is there something wrong with my script?

I’m expecting that when I run the .sh file, there should be the expected data like the above .txt RESULT. My main purpose creating this bash script is because AWS Cloudwatch Log insights has a limitation related to lambda , so i plan to create a bash script , using array in $lambda . This post is for my trial and error only. I need to make this work to move forward.

==========================

Tried the double quotation for $lambda and $queryId, same result no data. below

jomasangkay@DESKTOP-R1DM530:/mnt/c/Users/personal pc/Desktop/company/Bashscript/test$ bash -x testing2.sh
+ variables
+ lambdas=confidential
++ jq .queryId
++ awk -F '"' '{print $2}'
++ aws logs start-query --log-group-name confidential --start-time 1695704400 --end-time 1695707999 --query-string 'filter @message like "requestId" | parse @message "operationName: *," as API | parse @message "status: *," as Status | parse @message "channel: *," as Channel | stats count(*) as Total by API,Channel,Status | sort API,Total desc'
+ queryId=0f6c6fd6-da5e-40ed-9c74-ad22791455be
++ aws logs get-query-results --query-id 0f6c6fd6-da5e-40ed-9c74-ad22791455be
++ jq .
+ results='{
  "results": [],
  "statistics": {
    "recordsMatched": 0,
    "recordsScanned": 0,
    "bytesScanned": 0
  },
  "status": "Running"
}'
+ echo '{
  "results": [],
  "statistics": {
    "recordsMatched": 0,
    "recordsScanned": 0,
    "bytesScanned": 0
  },
  "status": "Running"
}'

===============================

Updated as of Oct 3, 2023 – 05:15 AM SINGAPORE TIME

I isolated the issue by creating a 2 .txt file called test1.txt and test2.txt , and their contents below,

test1.txt
c86c7f6b-95e4-4d17-9417-2f6987989316

test2.txt
c86c7f6b-95e4-4d17-9417-2f6987989316

As you can see above they have the exact same value (no spaces, no unusual text or whatsoever.)

See may new modified code below,

#!/bin/bash

variables () {
lambdas="confidential"

}
variables

queryId=$(aws logs start-query --log-group-name "$lambdas" --start-time 1695704400 --end-time 1695707999 --query-string 'filter @message like "requestId"' | jq '.queryId' | awk -F '"' '{print $2}')

#NOT WORKING
echo $queryId > test1.txt

#WORKING
echo c86c7f6b-95e4-4d17-9417-2f6987989316 > test2.txt


queryIds=$(awk '{print $1}' test2.txt)

results=$(aws logs get-query-results --query-id $queryIds | jq .)

echo "$results" > testextract.txt

When i assigned the variable from test1.txt, its not working no data extracted.

When i assigned the variable from test2.txt, its working as expected.

For some reason, looks like the value from test1.txt query id converted to somewhat special character, im not sure.