Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

cat and curl large JSON value from gitlab ci

There are many questions on how to post JSON via curl. And I am able to send one word as json but not the whole text.

My requirement is to comment a merge request when one job fail in our GitLab CI.

# From .gitlab-ci.yaml
  after_script:
    - report=$(find services/ -name '*_latest_failure.md')
    - comment="Job failed.nn$(cat "$report")"
    - if [[ "${CI_JOB_STATUS}" == "failed" ]] ; then bash -c "source .gitlab-ci/gitlab.sh; gitlab-comment_merge_request "$comment"" ; fi

# From .gitlab-ci/gitlab.sh
function gitlab-comment_merge_request() {
  url="https://$CI_SERVER_HOST/api/v4/projects/$CI_MERGE_REQUEST_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
  comment="$1"
  curl -v --location -X POST "$url" 
    --header "PRIVATE-TOKEN: $GITLAB_CI_TOKEN" 
    --header "Content-Type: application/json" 
    --data '{"body": "'"$comment"'"}'
}

Unfortunately, the comment that is created contains only the first word, Job.

I have tried many techniques to build and send the JSON payload. But all give the same result.

So I thought it could be a mistake when creating the $comment variable in my .gitlab-ci.yaml. However, echo $comment in the same file give the right content.

Now, I’m lost. Do you know why the $comment variable that should contain a large text (a markdown text with title, subtitles, and lists) is not sent ?

Thanks

Tags:

2

Answers


  1. 0

    A simplified version of this would look like:

    nl='
'
comment="Job failed.${nl}${nl}$(find services/ -name '*_latest_failure.md' -exec cat -- {} +)"
if [ "${CI_JOB_STATUS}" = "failed" ] ; then
  . .gitlab-ci/gitlab.sh
  gitlab_comment_merge_request "$comment"
fi

    Note:

    • Because you were quoting the output from find before passing it to cat, cat was looking for a single file with all the filenames concatenated together as its name. Don’t do that. Using find ... -exec cat -- {} + correctly handles names with spaces without creating new bugs.
    • We’ve taken out bash-only syntax: Changed == to =, [[ to [.
    • We took out the use of bash -c and the exec boundary between parent and child processes it implied. If we hadn’t done this we would need to export comment.
    • We changed the source keyword to its POSIX-compliant synonym, .
    • We took the dash out of the function name (this does require the file being sourced to change).

    A POSIX-compliant version of your function might look like:

    # note we had to rename this: dashes are not required to work in function names in sh
gitlab_comment_merge_request() {
  url="https://$CI_SERVER_HOST/api/v4/projects/$CI_MERGE_REQUEST_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
  comment="$1"
  body=$(jq -n --var comment "$1" '{"body":$comment}') 
  curl -v --location -X POST "$url" 
    --header "PRIVATE-TOKEN: $GITLAB_CI_TOKEN" 
    --header "Content-Type: application/json" 
    --data '{"body": "'"$comment"'"}'
}

    If you don’t have jq installed, it’s straightforward to write a shell wrapper for Python’s json module for the same purpose.

    Login or Signup to reply.
  2. 0

    It is possible to minimize all the logic and JSON formatting into a single shell script:

    gitlab-ci.yaml:

      after_script:
    - bash failed-reports.sh

    failed-reports.sh:

    #!/usr/bin/env sh

# If no failed job do nothing, exit
[ "$CI_JOB_STATUS" = failed ] || exit

# Dummy url for demo purpose
url='https://example.com/'

# Prepare report as a stream
{
    printf 'Job failed.nn'
    printf '%sn' ./services/*_latest_failure.md
} |
# Stream to jq to process report as raw input string into JSON
jq 
    --slurp 
    --raw-input 
    --compact-output 
    '{"body": .}' |
# Stream JSON data to curl so that it can handle very large payloads
curl 
    --url "$url" 
    --location 
    --request POST 
    --header "PRIVATE-TOKEN: $GITLAB_CI_TOKEN" 
    --header "Content-Type: application/json" 
    --data @- # This takes data from the STDIN stream

    In case the JSON data payload is very large, it is a safer option to stream it to curl with --data @- rather than passing it as an argument which turns out to be rather limited by the system’s maximum allowed argument length.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search