Question posted in
The official documentation can be found here.

Docker – Gitlab CI/CD not using npm cache

I have a monorepo app, with a folder "backend", which is an express API server. I’m using Gitlab (v17) CI/CD.

Here is the Dockerfile:

FROM node:alpine as installer
WORKDIR /app
COPY package*.json ./
RUN npm install --verbose

FROM node:lts-alpine as release
WORKDIR /app
COPY --from=installer /app /app
COPY . .

ENV PORT 3000
EXPOSE 3000

CMD [ "npm", "run", "start:production" ]

And here is a portion of my gitlab-ci.yml file:

stages:
  - build
  - deploy-stage
  - deploy-prod

variables:
  # VARIABLE_DATA: Gitlab-CI-YAML
  DOCKER_HOST: tcp://docker:2375 
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

build:
  stage: build
  image: docker:25.0.3-git
  services:
    - name: docker:25.0.3-dind
      alias: docker
  script:
    # get node app version from package.json for use in tagging docker image
    - apk update && apk add jq
    - export VERSION_BACKEND=`jq -r ".version" < ./backend/package.json`
    # login to docker
    - echo "$MY_REGISTRY_PASSWD" | docker login -u $MY_REGISTRY_USER $MY_REGISTRY_URL --password-stdin
    # build and tag docker image
    - docker build -t $MY_REGISTRY_URL/app-backend:$VERSION_BACKEND -t $MY_REGISTRY_URL/app-backend:latest -f ./backend/Dockerfile ./backend

    # publish finished image
    - docker push $MY_REGISTRY_URL/app-backend:$VERSION_BACKEND
    - docker push $MY_REGISTRY_URL/app-backend:latest
    # save version number to a file
    - echo $VERSION_BACKEND > version-backend.txt
  artifacts:
    paths:
      - version-backend.txt

My problem is that every time the pipeline is launched, the line npm install --verbose doesn’t take into account the cache, leading to a long wait for all the dependencies to be downloaded. I thought using multi-stage Dockerfile would solve the problem, bu I keep having a lot of lines like this:

#11 4.592 npm http fetch GET 200 https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.1.0.tgz 1935ms (cache miss)

What is strange is that I have some (not much…) lines indicating that the cache was not missed:

#11 2.488 npm http fetch GET 200 https://registry.npmjs.org/npm 658ms

I commit the package.json and package-lock.json files, as I read it would help the use of cache, up to no avail.

I’m open to any suggestion here.

Tags:

2

Answers


  1. 0

    Maybe the dependencies in package.json are devDependencies which always downloaded when run npm install command. That might be the issue please check those. Also docs in docs mentioned that

    1. Different jobs dependencies need to be same
    2. Define cache per job by using the cache keyword. Otherwise it is disabled.
      These method might work.
    Login or Signup to reply.
  2. 0

    My guess is that your infrastructure clears docker caches between runs or runs on different nodes.

    Since your local build cache seems to be unreliable you could try to cache to a remote backend:
    Docker Buildx Cache Backends
    This would allow you to share caches between pipelines running on different nodes/without shared cache.

    There is also some limited Gitlab Documentation on how to use the inline cache

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search