skip to Main Content

I would like to allow two jobs to run in parallel and both use the same environment. I have attached a sample github actions .yml file with comments to better explain what I need. I am new to the community so thank you for the help in advance!

I made a simplified code with the areas I need help with below.

name: Need two jobs to run in parallel and use the same `venv` from the first job

on: push

jobs:
  install-dependency:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11.9'

      - name: Install Dependencies
        run: |
          python -m venv venv
#         Save the VENV somehow!
          source venv/bin/activate
          pip install -r requirements.txt

  run-pytest:
    runs-on: ubuntu-latest
    needs: [install-dependency]
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v4

      - name: Run Pytest
        run: |
          source ${{ insert-venv-path }}/bin/activate
          pip install pytest
          pytest --maxfail=1 --disable-warnings -v

  run-other-pytest:
    runs-on: ubuntu-latest
    needs: [install-dependency]
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v4

      - name: Run Pytest
        run: |
          source ${{ insert-venv-path }}/bin/activate
          pip install pytest
          pytest --maxfail=1 --disable-warnings -v

I know github makes it so that every job runs on a different environment but I still thought this was possible. Please correct me if I am wrong.

2

Answers


  1. Chosen as BEST ANSWER

    The workflow above does work however it takes a very long time. Due to zipping and unzipping thousands of files in a venv.


  2. All jobs in a workflow run on a different and clean runner. This is to prevent polluting your environment, file system, etc by changes from other/previous jobs.

    The only way to share file system from install-dependency job is for it to upload a zip file of venv directory using

    - uses: actions/upload-artifact@v4
      with:
        name: venv
        path: ./venv
    

    and then download it in each of the other jobs and activate:

    - uses: actions/download-artifact@v4
      with:
        name: venv
        path: ./venv
    - name: Activate
      run: source venv/bin/activate
    

    Here is the complete workflow:

    name: venv
    
    on:
      workflow_dispatch:
    
    jobs:
      install-dependency:
        runs-on: ubuntu-latest
    
        steps:
          - name: Checkout code
            uses: actions/checkout@v4
    
          - name: Set up Python
            uses: actions/setup-python@v4
            with:
              python-version: '3.11.9'
    
          - name: Install Dependencies
            run: |
              python -m venv venv
              source venv/bin/activate
              python -m pip install requests
    
          - uses: actions/upload-artifact@v4
            with:
              name: venv
              path: ./venv
    
      run-pytest:
        runs-on: ubuntu-latest
        needs: [install-dependency]
        steps:
          - name: Checkout code
            uses: actions/checkout@v4
    
          - name: Setup Python
            uses: actions/setup-python@v4
            with:
              python-version: '3.11.9'
    
          - uses: actions/download-artifact@v4
            with:
              name: venv
              path: ./venv
    
          - name: Activate
            run: source venv/bin/activate
    
    
      run-other-pytest:
        runs-on: ubuntu-latest
        needs: [install-dependency]
        steps:
          - name: Checkout code
            uses: actions/checkout@v4
    
          - name: Setup Python
            uses: actions/setup-python@v4
            with:
              python-version: '3.11.9'
    
          - uses: actions/download-artifact@v4
            with:
              name: venv
              path: ./venv
    
          - name: Activate
            run: source venv/bin/activate
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search