skip to Main Content

Summary

What specific syntax must be changed in the code below in order for the multi-line contents of the $MY_SECRETS environment variable to be 1.) successfully written into the C:\Users\runneradmin\somedir\mykeys.yaml file on a Windows runner in the GitHub workflow whose code is given below, and 2.) read by the simple Python 3 main.py program given below?

PROBLEM DEFINITION:

The echo "$MY_SECRETS" > C:\Users\runneradmin\somedir\mykeys.yaml command is only printing the string literal MY_SECRETS into the C:\Users\runneradmin\somedir\mykeys.yaml file instead of printing the multi-line contents of the MY_SECRETS variable.

We confirmed that this same echo command does successfully print the same multi-line secret in an ubuntu-latest runner, and we manually validated the correct contents of the secrets.LIST_OF_SECRETS environment variable. … This problem seems entirely isolated to either the windows command syntax, or perhaps to the windows configuration of the GitHub windows-latest runner, either of which should be fixable by changing the workflow code below.

EXPECTED RESULT:

The multi-line secret should be printed into the C:\Users\runneradmin\somedir\mykeys.yaml file and read by main.py.

The resulting printout of the contents of the C:\Users\runneradmin\somedir\mykeys.yaml file should look like:

***  
***  
***  
***  

LOGS THAT DEMONSTRATE THE FAILURE:

The result of running main.py in the GitHub Actions log is:

ccc item is:  $MY_SECRETS

As you can see, the string literal $MY_SECRETS is being wrongly printed out instead of the 4 *** secret lines.

REPO FILE STRUCTURE:

Reproducing this error requires only 2 files in a repo file structure as follows:

.github/
    workflows/
        test.yml
main.py   

WORKFLOW CODE:

The minimal code for the workflow to reproduce this problem is as follows:

name: write-secrets-to-file
on:
  push:
    branches:
      - dev
jobs:
  write-the-secrets-windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v3
      - shell: python
        name: Configure agent
        env:
          MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
        run: |
          import subprocess
          import pathlib
          pathlib.Path("C:\Users\runneradmin\somedir\").mkdir(parents=True, exist_ok=True)
          print('About to: echo "$MY_SECRETS" > C:\Users\runneradmin\somedir\mykeys.yaml')
          output = subprocess.getoutput('echo "$MY_SECRETS" > C:\Users\runneradmin\somedir\mykeys.yaml')
          print(output)
          os.chdir('D:\a\myRepoName\')
          mycmd = "python myRepoName\main.py"
          p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
          while(True):
            # returns None while subprocess is running
            retcode = p.poll() 
            line = p.stdout.readline()
            print(line)
            if retcode is not None:
              break 

MINIMAL APP CODE:

Then the minimal main.py program that demonstrates what was actually written into the C:\Users\runneradmin\somedir\mykeys.yaml file is:

with open('C:\Users\runneradmin\somedir\mykeys.yaml') as file:
  for item in file:
    print('ccc item is: ', str(item))
    if "var1" in item:
      print("Found var1")

STRUCTURE OF MULTI-LINE SECRET:

The structure of the multi-line secret contained in the secrets.LIST_OF_SECRETS environment variable is:

var1:value1
var2:value2
var3:value3
var4:value4

These 4 lines should be what gets printed out when main.py is run by the workflow, though the print for each line should look like *** because each line is a secret.

4

Answers


  1. You need to use yaml library:

    import yaml
    
    data = {'MY_SECRETS':'''
    var1:value1
    var2:value2
    var3:value3
    var4:value4
    '''}#add your secret 
    
    with open('file.yaml', 'w') as outfile: # Your file
        yaml.dump(data, outfile, default_flow_style=False)
    

    This is result:
    Result
    I used this.

    Login or Signup to reply.
  2. I tried the following code and it worked fine :

    LIST_OF_SECRETS

    key1:val1
    key2:val2
    

    Github action (test.yml)

    name: write-secrets-to-file
    on:
      push:
        branches:
          - main
    jobs:
      write-the-secrets-windows:
        runs-on: windows-latest
        steps:
          - uses: actions/checkout@v3
          - shell: python
            name: Configure agentt
            env:
              MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
            run: |
              import base64, subprocess, sys
              import os
              secrets = os.environ["MY_SECRETS"]
              
              def powershell(cmd, input=None):
                  cmd64 = base64.encodebytes(cmd.encode('utf-16-le')).decode('ascii').strip()
                  stdin = None if input is None else subprocess.PIPE
                  process = subprocess.Popen(["powershell.exe", "-NonInteractive", "-EncodedCommand", cmd64], stdin=stdin, stdout=subprocess.PIPE)
                  if input is not None:
                      input = input.encode(sys.stdout.encoding)
                  output, stderr = process.communicate(input)
                  output = output.decode(sys.stdout.encoding).replace('rn', 'n')
                  return output
              
              command = r"""$secrets = @'
              {}
              '@
              $secrets | Out-File -FilePath .mykeys.yaml""".format(secrets)
              
              command1 = r"""Get-Content -Path .mykeys.yaml"""
              
              powershell(command)
              print(powershell(command1))
    

    Output

    ***
    ***
    

    As you also mention in the question, Github will obfuscate any printed value containing the secrets with ***

    EDIT : Updated the code to work with multiple line secrets. This answer was highly influenced by this one

    Login or Signup to reply.
  3. Edit: updated with fixed main.py and how to run it.

    You can write the key file directly with Python:

          - shell: python
            name: Configure agent
            env:
              MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
            run: |
              import os
              import pathlib
              pathlib.Path('C:\Users\runneradmin\somedir\').mkdir(parents=True, exist_ok=True)
              with open('C:\Users\runneradmin\somedir\mykeys.yaml', 'w') as key_file:
                key_file.write(os.environ['MY_SECRETS'])
          - uses: actions/checkout@v3
          - name: Run main
            run: python main.py
    

    To avoid newline characters in your output, you need a main.py that removes the newlines (here with .strip().splitlines()):

    main.py

    with open('C:\Users\runneradmin\somedir\mykeys.yaml') as file:
        for item in file.read().strip().splitlines():
            print('ccc item is: ', str(item))
            if "var1" in item:
                print("Found var1")
    

    Here’s the input:

    LIST_OF_SECRETS = '
    key:value
    key2:value
    key3:value
    '
    

    And the output:

    ccc item is:  ***
    Found var1
    ccc item is:  ***
    ccc item is:  ***
    ccc item is:  ***
    

    Here is my complete workflow:

    name: write-secrets-to-file
    on:
      push:
        branches:
          - master
    jobs:
      write-the-secrets-windows:
        runs-on: windows-latest
        steps:
          - shell: python
            name: Configure agent
            env:
              MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
            run: |
              import os
              import pathlib
              pathlib.Path('C:\Users\runneradmin\somedir\').mkdir(parents=True, exist_ok=True)
              with open('C:\Users\runneradmin\somedir\mykeys.yaml', 'w') as key_file:
                key_file.write(os.environ['MY_SECRETS'])
          - uses: actions/checkout@v3
          - name: Run main
            run: python main.py
    

    Also, a simpler version using only Windows shell (Powershell):

          - name: Create key file
            env:
              MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
            run: |
              mkdir C:\Users\runneradmin\somedir
              echo "$env:MY_SECRETS" > C:\Users\runneradmin\somedir\mykeys.yaml
          - uses: actions/checkout@v3
          - name: Run main
            run: python main.py
    
    Login or Signup to reply.
  4. The problem is – as it is so often – the quirks of Python with byte arrays and strings and en- and de-coding them in the right places…

    Here is what I used:

    test.yml:

    name: write-secrets-to-file
    on:
      push:
        branches:
        - dev
    jobs:
      write-the-secrets-windows:
        runs-on: windows-latest
        steps:
          - uses: actions/checkout@v3
          - shell: python
            name: Configure agent
            env:
              MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
            run: |
              import subprocess
              import pathlib
              import os
              # using os.path.expanduser() instead of hard-coding the user's home directory
              pathlib.Path(os.path.expanduser("~/somedir")).mkdir(parents=True, exist_ok=True)
              secrets = os.getenv("MY_SECRETS")
              with open(os.path.expanduser("~/somedir/mykeys.yaml"),"w",encoding="UTF-8") as file:
                  file.write(secrets)
              mycmd = ["python","./main.py"]
              p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
              while(True):
                  # returns None while subprocess is running
                  retcode = p.poll()
                  line = p.stdout.readline()
                  # If len(line)==0 we are at EOF and do not need to print this line.
                  # An empty line from main.py would be 'n' with len('n')==1!
                  if len(line)>0:
                    # We decode the byte array to a string and strip the
                    # new-line characters r and n from the end of the line,
                    # which were read from stdout of main.py
                    print(line.decode('UTF-8').rstrip('rn'))
                  if retcode is not None:
                      break
    

    main.py:

    import os
    # using os.path.expanduser instead of hard-coding user home directory
    with open(os.path.expanduser('~/somedir/mykeys.yaml'),encoding='UTF-8') as file:
        for item in file:
            # strip the new-line characters r and n from the end of the line
            item=item.rstrip('rn')
            print('ccc item is: ', str(item))
            if "var1" in item:
                print("Found var1")
    

    secrets.LIST_OF_SECRETS:

    var1: secret1
    var2: secret2
    var3: secret3
    var4: secret4
    

    And my output in the log was

    ccc item is:  ***
    Found var1
    ccc item is:  ***
    ccc item is:  ***
    ccc item is:  ***
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search