Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Ubuntu – github actions workflow – ssh into server, download single file via curl, unable to open file

I have a github workflow which is supposed to login via SSH into a remote server and then download the docker-compose file from a private github repository.

I was first struggling to get the multi-line string syntax right for the run section in the workflow yaml file, in particular the curl command, since it requires quotes for the header. The solution i found is to split the curl command into environment variables.

download-file:
    name: download file
    runs-on: ubuntu-latest

    steps:
      - name: install ssh keys
        # check this thread to understand why its needed:
        # <https://stackoverflow.com/a/70447517>
        run: |
          install -m 600 -D /dev/null ~/.ssh/id_rsa
          echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
          host='${{ secrets.SSH_HOST }}'
          hosts="$(dig +short "$host" | grep -v '.$' | sed -z 's|n|,|g')$host"
          ssh-keyscan -H "$hosts" > ~/.ssh/known_hosts

      - name: download docker compose file from private repo to server
        run: ssh ${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }}
          # the first curl call to the public repo works. 
          # the file can be opened. the second curl call also 
          # finishes but the file can NOT be opened. 
          "ls -la
          && cd test
          && ls -la
         
          && curl curl $CURL_PUBLIC_URL -o $URL_PUBLIC_O
          && curl -v -H $CURL_H1 -H $CURL_H2 -H $CURL_H3 -o $CURL_O $CURL_URL
          && exit"

        env:
          CURL_H1: '""Authorization: Bearer ${{secrets.PULL_ACCESS_TOKEN}}""'
          CURL_H2: '""Accept: application/vnd.github.v3.raw""'
          CURL_H3: '""Content-Type: text/html; charset=UTF-8""'
          CURL_O: "docker-compose.yml"
          CURL_URL: "https://raw.githubusercontent.com/<username>/<repository name>/refs/heads/main/docker-compose.prod.yml"
          CURL_PUBLIC_URL: "https://raw.githubusercontent.com/thnk2wn/rasp-cat-siren/master/siren/Dockerfile"
          CURL_PUBLIC_O: "Dockerfile"

      - name: cleanup
        run: rm -rf ~/.ssh

PROBLEM:

When I execute the curl command after logging in manually into the server, the file is downloaded and I can open it, no problem.

Within the github workflow, the curl request also executes without any error, but when I try to open the downloaded file docker-compose.yml via cat docker-compose.yml after ssh manually into the server, the file can not be opened and the message found an invalid character in header name is displayed instead.

My guess is that there is some kind of issue with the string formatting of the variables. It already took me hours experimenting with various string formats escaping the double quote with the backslash to keep the double quote. The string syntax looks already quite overly complicated to me to be honest.

EDIT 1: When I use wget instead of curl in the actions run command, then i get a 400 response. When I execute the wget request manually on the server, the file is downloaded without any issue.

EDIT 2: Downloading a file from a public github repository works without any problems and can be opened with no issues.

QUESTION:

How can I run the curl request after ssh into the server downloading the single docker-compose file from the private github repo without corrupting the file so that the file can actually be opened?
Respectively, is there an issue with the formatting of the syntax for the curl command within runconsidering the string formatting of the envvariables and if so, what would be a better/ correct way?

Thanks for your help!

Tags:

3

Answers


  1. Chosen as BEST ANSWER
    0

    It seems the culprit was somehow the header information in the curl request. So I found this solution where the access token is placed inside the URL. The critical piece here is the syntax of the env variable CURL_URL:

    # ...
download-file:
    name: download file
    runs-on: ubuntu-latest

    env:
      CURL_O: "docker-compose.yml"
      CURL_PATH: "raw.githubusercontent.com/<username>/<repository name>/refs/heads/main/docker-compose.prod.yml"

    steps:
      - name: install ssh keys
        # check this thread to understand why its needed:
        # <https://stackoverflow.com/a/70447517>
        run: |
          install -m 600 -D /dev/null ~/.ssh/id_rsa
          echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
          host='${{ secrets.SSH_HOST }}'
          hosts="$(dig +short "$host" | grep -v '.$' | sed -z 's|n|,|g')$host"
          ssh-keyscan -H "$hosts" > ~/.ssh/known_hosts

      - name: test pull single file from private repo
        env:
          CURL_URL: "https://${{secrets.PULL_ACCESS_TOKEN}}@${{env.CURL_PATH}}"
        run: |
          ssh ${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }} <<'ENDSSH'
          cd test
          ls -la
          curl ${{env.CURL_URL}} -o ${{env.CURL_O}}
          ENDSSH

      - name: cleanup
        run: rm -rf ~/.ssh

  2. 0

    Here’s why this causes issues:

    • When you run ssh user@host "command with $VARIABLE", the $VARIABLE is evaluated on the local machine (the runner), not on the remote machine. However, since you’re enclosing the command in double quotes, the local shell doesn’t substitute those variables because they are not set locally (they’re meant for the remote machine).
    • Once connected, the remote shell tries to interpret $CURL_H1, but since those variables aren’t defined on the remote server’s environment, they are empty, leading to incorrect commands being executed.

    To fix this issue, you need to:

    • Ensure that the variables are expanded before they are sent over SSH.
    • Avoid relying on environment variables being available in the remote session unless you explicitly pass them.

    Explanation:

    • We define all variables within the run block, ensuring they are available in the runner’s shell.
    • The curl command is constructed as a string in CURL_CMD, with proper quoting to handle spaces and special characters.
    • When we run the SSH command, the variables are expanded on the runner before the command is sent to the remote server.
    • The entire command within the double quotes "…" in the SSH command is what gets executed on the remote server.
    • By constructing the command this way, the remote server receives a fully-formed command with all variables expanded, eliminating the issue of missing environment variables.
    Login or Signup to reply.
  3. 0

    git example, maybe need some minor to adjust env names

    cat $sshKey > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
# ssh-keyscan -t rsa,dsa github.com >> ~/.ssh/known_hosts

git archive --format=tar [email protected] "+YOUR-BRANCH+" docker-compose.prod.yml | tar -O -xf - > docker-compose.yml

    Note: I don’t know what you’ll do next. But it’s never a good idea to use docker-compose in prod.
    It’s just to start, you need to avoid it when your project grow up

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search