I’m attempting to build a Next.js project on CircleCI using an ARM-based machine (specifically, arm.xlarge
) which has 32GB of RAM. However, it seems to very quickly exit with an error about lack of memory. I’ve tried to increase the memory of the Node process all the way up to that of the machine, but with no luck – I receive the same error.
What’s particularly strange is that the GCs have a tiny amount of memory utilization (3.5MB), and that it exits in sub-second time!
This build does work, if I swap out the arm
resource type for an x86_64
resource type, and build a Docker image for the x86_64
platform instead. Obviously this doesn’t cover my needs, but verifies the build works.
What might cause this very strange behaviour? What else can I do to further debug this issue?
What I’ve tried
- Changing the Node version, ranging from 14 to 18.8.0
- Changing the machine resource size
- Changing the options of Docker, including adding a
--memory
flag to theextra-build-args
field of the build job - Removing the
yarn.lock
- Purging the Node cache
- Deleting the
node_modules
folder
Relevant resources:
Here’s the trace:
#16 [11/11] RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build
#16 sha256:da0777711cb35943386ef4812f2df1ff4932fb5bee717be330a217104e5240c7
#16 0.178
#16 0.178 <--- Last few GCs --->
#16 0.178
#16 0.178 [1:0x570e320] 100 ms: Mark-sweep 1.2 (3.5) -> 1.2 (3.5) MB, 2.2 / 0.0 ms (average mu = 0.818, current mu = 0.007) allocation failure scavenge might not succeed
#16 0.178 [1:0x570e320] 102 ms: Mark-sweep (reduce) 1.2 (3.5) -> 1.2 (3.5) MB, 2.3 / 0.0 ms (average mu = 0.704, current mu = 0.006) last resort GC in old space requested
#16 0.178 [1:0x570e320] 104 ms: Mark-sweep (reduce) 1.2 (2.5) -> 1.1 (3.5) MB, 2.3 / 0.0 ms (average mu = 0.549, current mu = 0.010) last resort GC in old space requested
#16 0.178
#16 0.178
#16 0.178 <--- JS stacktrace --->
#16 0.178
#16 0.178 FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
#16 ERROR: process "/bin/sh -c NODE_OPTIONS="--max_old_space_size=8192" yarn build" did not complete successfully: exit code: 132
------
> [11/11] RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build:
------
Dockerfile:32
--------------------
31 |
32 | >>> RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build
34 |
--------------------
error: failed to solve: rpc error: code = Unknown desc = process "/bin/sh -c NODE_OPTIONS="--max_old_space_size=8192" CI_BUILD=${CI_BUILD} DEPLOYMENT_VERSION=${DEPLOYMENT_VERSION} yarn build" did not complete successfully: exit code: 132
Here’s the relevant parts of the Dockerfile:
FROM node:14.18.0-alpine
WORKDIR /app
RUN apk update
RUN apk add python3
RUN apk add git gnupg jq g++ make py3-pip
COPY package.json yarn.lock ./
RUN yarn
RUN yarn add [email protected] --global
COPY . .
RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build
EXPOSE 3001
CMD [ "npx", ... ]
and here’s the job from CircleCI config.yml
:
build_and_push_image:
machine:
image: ubuntu-2004:202101-01
resource_class: arm.xlarge
steps:
- checkout
- docker/pull:
images: 'node:16.14.2'
- aws-ecr/build-and-push-image:
aws-access-key-id: AWS_ACCESS_KEY_ID
aws-secret-access-key: AWS_SECRET_ACCESS_KEY
aws-cli-version: latest
create-repo: true
skip-when-tags-exist: true
dockerfile: Dockerfile
extra-build-args: >
--build-arg GPG_ENCRYPT_PASSPHRASE
--build-arg BUILD_STAGE
--build-arg CI_BUILD=true
platform: linux/arm64/v7
push-image: true
registry-id: AWS_ACCOUNT_ID
region: $AWS_REGION
repo: $PROJECT_NAME
tag: $version_tag
2
Answers
After a good few hours of debugging with trial and error, I fixed with the following changes to
config.yml
:I'm not sure why this works, but here's my best thoughts for others who are blocked by this in the future:
docker buildx
command from the CircleCI orb. I'm also noticing that a lot of people are having problems with Docker image caching using the orb, and maybe there's some issue with the command itself (see https://github.com/CircleCI-Public/aws-ecr-orb/issues/202). It looks something like this:qemu
dependencieslibseccomp
blocking unknown (new) syscallsHave you checked the versions of OS and Node? The Versions should match. (x64=x64)