Question posted in
The official documentation can be found here.

Combine several docker images to avoid long build times

I have a docker image being built as a multi stage build

from ubuntu:latest as base

# do some stuff

from base as llvm 

# build llvm (takes ~1.5 hours)

from llvm as independent_big_project

# build independent big project (takes ~1 hours)

I build llvm which takes a long time, and then another large project which is not dependent on llvm. I may have to tweak the settings for both llvm and my other project. If I put llvm first I will have to rebuild the other project as it is based on the previous stage, if I build the project first then I will have to rebuild llvm for the same reason.

Is there a way to split this into independent sections and combine both projects into one image at the end?

I could use multiple docker images and copy the completed builds across, but I think this would need to be managed manually?

Is there a way to automate this process where I can just run docker build on a top level image, and the dependencies are built if required, but separately?

Tags:

2

Answers


  1. 0

    Consider copy --from, along:

    from ubuntu:latest as base

from base as llvm 

from base as independent_big_project

from ubuntu:latest as the_result
copy --from independent_big_project /the/result /
copy --from llvm /the/result /
    Login or Signup to reply.
  2. 0

    You need to COPY --from files from one build stage to another. If you’re using the current BuildKit Docker image builder, it will try to build all of the build stages in parallel, and only block at the point where it wants to execute that COPY.

    For C and C++ programs, both the Autoconf and CMake tool sets generally support building a program to install into an arbitrary filesystem prefix. That will give you a self-contained directory path that you can COPY into later image. LLVM uses CMake and you should be able to do something like

    FROM ubuntu:24.04 AS base
...

FROM base AS llvm
RUN apt-get update 
 && DEBIAN_FRONTEND=noninteractive
    apt-get install --no-install-recommends --assume-yes 
      build-essential 
      cmake 
      git-core 
      libz-dev 
      ninja-build

RUN git clone --depth 1 https://github.com/llvm/llvm-project.git
WORKDIR /llvm-project
RUN cmake -S llvm -B build -G Ninja 
    -DCMAKE_INSTALL_PREFIX=/opt/llvm  # <--
RUN cmake --build build

FROM base AS independent_big_project
...

FROM base AS final
COPY --from=llvm /opt/llvm /opt/llvm

    Calling out the three important things from this fragment:

    1. We build LLVM with -DCMAKE_INSTALL_PREFIX=/opt/llvm so that it installs into its own filesystem tree.
    2. The second build stage, with the independent build, is FROM base; it does not depend on the LLVM build.
    3. We COPY --from=llvm /opt/llvm /opt/llvm in the final build stage. It’s often important to use the same filesystem path when copying install trees like this.

    You’ll probably need to similarly COPY --from=independent_big_project. If LLVM is installing C shared libraries, you may need to set ENV LD_LIBRARY_PATH=/opt/llvm/lib or to append to /etc/ld.so.conf in the final build image so those can be used at runtime.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search