Question 246775 in Docker Question posted in
The official documentation can be found here.

Docker – Use previous working directory in current job

I currently have a single job that does everything (testing, building and Docker imaging).
I would like to move from this single job workflow to a multi job workflow in order to parallelise tasks.

Currently it consists of: Testing –> Code Building –> Docker Building (3 images built sequentially)

I would like to move to this paradigm:

  1. Testing –> Code Building
  2. Docker Building (3 images built in parallel)

The reason why I went with the first system is because until now I haven’t found a way to persist the installed packages and built artifacts across jobs. The first two steps generate the node_modules and the dist folders.

Is there a way to do so? Am I missing something?

If not, is there a smarter alternative?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    @VonC posted a great answer, I'm just adding a few minor details to the advancements I've made.

    As he said, artifacts provide an excellent way to transfer data.

    Upload them when you're saving them:

    - uses: actions/upload-artifact@v3
  with:
    name: apps-dist
    path: |
      ${{ github.workspace }}/dist/**

    Download them when you need them

    - name: Download built dist
  uses: actions/download-artifact@v3
  with:
    name: apps-dist
    path: '${{ github.workspace }}/dist'

    Parallelisation of multiple docker builds is also rather simple, instead of going thru Dockerfiles like he proposed, I took a more programmatic approach:

    Build-Docker-Images:
  if: '${{ github.event_name != ''pull_request'' }}'
  needs: Test-Build-Apps
  permissions:
    id-token: write
    contents: write
    actions: read
    packages: write
  strategy:
    matrix:
      image_name:
        - app-a
        - app-b
        - app-c

    And everytime I needed to set metadata for tagging, I took this approach:

    - name: 'Docker meta ${{ matrix.image_name }}}'
  id: meta-data
  uses: docker/metadata-action@v4
  with:
    images: |
      ghcr.io/${{github.repository_owner}}/apps/${{ matrix.image_name }}
    tags: |
      type=schedule
      type=ref,event=branch
      type=ref,event=pr
      type=semver,pattern={{version}}
      type=semver,pattern={{major}}.{{minor}}
      type=semver,pattern={{major}}
      type=sha
- name: 'Build and push ${{ matrix.image_name }} Docker image'
  uses: docker/build-push-action@v4
  with:
    context: .
    file: './apps/${{ matrix.image_name }}/Dockerfile'
    push: true
    tags: '${{ steps.meta-data.outputs.tags }}'
    labels: >
      org.opencontainers.image.source=https://github.com/${{ github.repository
      }}
    cache-from: type=gha
    cache-to: 'type=gha,mode=max'

  2. 0

    In GitHub Actions, jobs can run on different runners and do not share the file system, which is why data generated during one job is not available in subsequent jobs.

    To share data between jobs, you will have to use artifacts.

    For instance, here is a multi-job system workflow, where the "node_modules" and "dist" folders are shared across jobs:

    In the first job, perform your testing and code building as you currently do.
    After building the code, upload the "node_modules" and "dist" folders as artifacts. You will use the actions/upload-artifact action to upload these folders.

    jobs:
    build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
        uses: actions/checkout@v4

    - name: Install dependencies and build
        run: |
        npm install
        npm run build

    - name: Upload node_modules
        uses: actions/upload-artifact@v3
        with:
        name: node_modules
        path: node_modules

    - name: Upload dist
        uses: actions/upload-artifact@v3
        with:
        name: dist
        path: dist

    In the subsequent Docker building job, you will first download the artifacts using the actions/download-artifact action and then build your Docker images.

        docker_build:
    needs: build
    runs-on: ubuntu-latest

    strategy:
        matrix:
        dockerfile: [Dockerfile1, Dockerfile2, Dockerfile3]

    steps:
    - name: Checkout code
        uses: actions/checkout@v2

    - name: Download node_modules
        uses: actions/download-artifact@v2
        with:
        name: node_modules
        path: node_modules

    - name: Download dist
        uses: actions/download-artifact@v2
        with:
        name: dist
        path: dist

    - name: Build Docker image
        run: |
        docker build -f ${{ matrix.dockerfile }} -t my_image .

    This is a build matrix for the Docker building job to build three Docker images in parallel, using three different Dockerfiles named "Dockerfile1", "Dockerfile2", and "Dockerfile3".

    This workflow allows you to parallelize your Docker image builds while sharing the "node_modules" and "dist" folders across jobs, which were generated in the initial build job.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search