TimurShtatland
June 21, 2022
256 views
0 votes
2 Answers

I am setting up continuous integration using GitHub Actions. One of the prerequisites (samtools) is most easily installed by conda. The standard way to use installed packages is to activate the corresponding conda environment. I am looking for a way to activate the conda environment. The common methods to activate it failed, see details below. My current workaround is to add to the PATH a hardcoded path to samtools, installed by conda. But this is not maintainable if the number of installed packages increases. It is also not the standard way to use packages installed with conda.

DETAILS:

.github/workflows/conda.yml

name: Conda
on: [push]

jobs:
  # label of the job
  tests:
    name: Tests
    # containers must run in Linux based operating systems
    runs-on: ubuntu-latest
    # Docker Hub image that `postgres-job` executes in
    container: node:latest
    # service containers to run with `postgres-job`
    steps:
      - uses: conda-incubator/setup-miniconda@v2
        with:
          miniconda-version: "latest"
          channels: bioconda, conda-forge, defaults
          use-only-tar-bz2: true  # IMPORTANT: This needs to be set for caching to work properly!
          auto-update-conda: true
          auto-activate-base: true
      - name: Install samtools
        run: |
            set -x

            echo "begin: PATH=$PATH;"

            conda create -y --name samtools samtools

            conda activate samtools || true
            echo "after conda activate samtools: PATH=$PATH;"
            which samtools || true
              
            $CONDA/bin/activate samtools || true
            echo "after CONDA/bin/activate samtools: PATH=$PATH;"
            which samtools || true
              
            export PATH="3/envs/samtools/bin:$PATH"
            echo "after export PATH=3/envs/samtools/bin:PATH: PATH=$PATH;"
            which samtools || true

Output:

Run set -x
begin: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ echo begin: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ conda create -y --name samtools samtools
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
[...]

# To activate this environment, use
#
#     $ conda activate samtools
#
# To deactivate an active environment, use
#
#     $ conda deactivate
+ conda activate samtools
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
    $ conda init <SHELL_NAME>
Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
+ true
after conda activate samtools: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ echo after conda activate samtools: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ which samtools
+ true
+ 3/bin/activate samtools
+ echo after CONDA/bin/activate samtools: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ which samtools
after CONDA/bin/activate samtools: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ true
after export PATH=3/envs/samtools/bin:PATH: PATH=3/envs/samtools/bin:3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ export PATH=3/envs/samtools/bin:3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ echo after export PATH=3/envs/samtools/bin:PATH: PATH=3/envs/samtools/bin:3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ which samtools
3/envs/samtools/bin/samtools

Answers

Chosen as BEST ANSWER

Add this to not ignore bash profile files (fix from: https://github.com/marketplace/actions/setup-miniconda ):

    defaults:
      run:
        shell: bash -l {0}

Complete GitHub Action yml file, .github/workflows/conda.yml:

name: Conda
on: [push]

jobs:
  # label of the job
  tests:
    name: Tests
    # containers must run in Linux based operating systems
    runs-on: ubuntu-latest
    # Do not ignore bash profile files. From:
    # https://github.com/marketplace/actions/setup-miniconda
    defaults:
      run:
        shell: bash -l {0}
    # Docker Hub image that `postgres-job` executes in
    container: node:latest
    # service containers to run with `postgres-job`
    steps:
      - uses: conda-incubator/setup-miniconda@v2
        with:
          miniconda-version: "latest"
          channels: bioconda, conda-forge, defaults
          use-only-tar-bz2: true  # IMPORTANT: This needs to be set for caching to work properly!
          auto-update-conda: true
          auto-activate-base: true
      - name: Install samtools
        run: |
            echo "begin: PATH=$PATH;"

            conda create -y --name samtools samtools

            conda activate samtools || true
            echo "after conda activate samtools: PATH=$PATH;"
            which samtools || true
            # Use samtools here.

Output:

begin: PATH=/__w/ngs-aggregate_results/ngs-aggregate_results/3/envs/test/bin:/__w/ngs-aggregate_results/ngs-aggregate_results/3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
...
after conda activate samtools: PATH=/__w/ngs-aggregate_results/ngs-aggregate_results/3/envs/samtools/bin:/__w/ngs-aggregate_results/ngs-aggregate_results/3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
/__w/ngs-aggregate_results/ngs-aggregate_results/3/envs/samtools/bin/samtools

(Edit)

Automatic Activation

The documentation also gives examples of automatic activation. Specifically, Example #3 demonstrates this with a couple key components:

As noted in the IMPORTANT section of the documentation and in the other answer, the shell needs to use login mode, which can be set globally within a job, using

jobs:
  example-3:
    defaults:
      run:
        shell: bash -l {0}

An environment definition is provided to setup-miniconda GHA. In the example it’s etc/example-environment.yml, which defines the environment anaconda-client-env and this is set to activate using the activate-environment argument.

    steps:
      - uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: anaconda-client-env
          environment-file: etc/example-environment.yml
          auto-activate-base: false
      - run: |
          conda info
          conda list

Samtools Example

I have a repository where I test environment definitions, so here’s an explicit example for samtools. Note that I prefer Mamba, and also recommend capturing an explicit mamba env export to document the environment.

envs/so-samtools.yaml

name: so-samtools
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - samtools

.github/workflows/so-samtools.yaml

name: so-samtools
on:
  push:
    paths:
      - 'envs/so-samtools.yaml'
      - '.github/workflows/so-samtools.yaml'

jobs:
  create-env:
    name: ${{ matrix.os }} 
    runs-on: ${{ matrix.os }}
    defaults:
      run:
        shell: bash -l {0}
        
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
        
    steps:
      - name: checkout repository
        uses: actions/checkout@v2
        
      - name: create environment with mamba
        uses: conda-incubator/setup-miniconda@v2
        with:
          mamba-version: "*"
          channels: conda-forge,bioconda,defaults
          auto-activate-base: false
          activate-environment: so-samtools
          environment-file: envs/so-samtools.yaml
      
      - name: check solution
        run: |
          mamba env export
      
      - name: test samtools
        run: |
          which samtools
          samtools help

Please signup or login to give your own answer.

Click here to cancel reply.

Ubuntu – How to activate conda environment in GitHub Actions?

Answers

Automatic Activation

Samtools Example