Question 81805 in Docker Question posted in
The official documentation can be found here.

How to seperate COPY to multiple steps for caching in Dockerfile

I’m building an Docker image with big files (>1.0GB) and small python scripts. Big files are rarely changed, so I want to caching it.

The directory is looks like:

- app/
  - main.py
  - modules/
      - foo.py
      - bar.py
  - big_files/
      - bigone.tar
      - bigtwo.tar

My first Dockerfile:

FROM python3:latest
COPY ./app /opt/app

When I update python scripts, it have to COPY all files which consume a long time.

What I want to acheive:

FROM python3:latest
COPY ./app/big_files /opt/app/big_files
COPY ./app /opt/app

However, it also copy big files too.

How to COPY in two step for caching?

Tags:

2

Answers


  1. 0

    You should change your app’s structure so that the big files are outside the /app folder.

    If you don’t want to do that, then you have to explicitly copy the files under /app:

    FROM python3:latest
COPY ./app/big_files /opt/app/big_files
COPY ./app/*.py /opt/app/
COPY ./app/modules /opt/app/
    Login or Signup to reply.
  2. 0

    Each step is happening in its own layer. In your case that means

    1. Pull base image
    2. Copy Big Files
    3. Copy ./app

    Everything must be run on the initial build.

    With a setup similar to yours:

    $ tree .
.
├── Dockerfile
└── app
    ├── bigFiles
    │   └── bigFile
    └── smallFile

    and a dockerfile:

    FROM python:latest

COPY ./app/bigFiles .
COPY ./app .

RUN echo 'Done'

    Now if you run the exact same build again and nothing has changed, each layer can be reused:

    $ docker build -t stackoverflowtest .

[+] Building 1.4s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 36B                                                                                                                                                                                                                                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/python:latest                                                                                                                                                                                                                                                                                                                                                                       1.3s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                                                                                                                                                                                                                                                                                                          0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 124B                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => [1/4] FROM docker.io/library/python:latest@sha256:e9c35537103a2801a30b15a77d4a56b35532c964489b125ec1ff24f3d5b53409                                                                                                                                                                                                                                                                                                                 0.0s
 => CACHED [2/4] COPY ./app/bigFiles .                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => CACHED [3/4] COPY ./app .                                                                                                                                                                                                                                                                                                                                                                                                          0.0s
 => CACHED [4/4] RUN echo 'Done'                                                                                                                                                                                                                                                                                                                                                                                                       0.0s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                                                                                                                                                0.0s
 => => writing image sha256:a78f5f730a893ccaaff102a7fe179461ec15c53bcbac926cb818e95d9d012875                                                                                                                                                                                                                                                                                                                                           0.0s
 => => naming to docker.io/library/stackoverflowtest

    If you touch the big files, that layer has to be rebuilt:

    $ echo "Change me" > app/bigFiles/bigFile
$ docker build -t stackoverflowtest .
[+] Building 0.9s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 36B                                                                                                                                                                                                                                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/python:latest                                                                                                                                                                                                                                                                                                                                                                       0.6s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 146B                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => CACHED [1/4] FROM docker.io/library/python:latest@sha256:e9c35537103a2801a30b15a77d4a56b35532c964489b125ec1ff24f3d5b53409                                                                                                                                                                                                                                                                                                          0.0s
 => [2/4] COPY ./app/bigFiles .                                                                                                                                                                                                                                                                                                                                                                                                        0.0s
 => [3/4] COPY ./app .                                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => [4/4] RUN echo 'Done'                                                                                                                                                                                                                                                                                                                                                                                                              0.2s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                                                                                                                                                0.0s
 => => writing image sha256:912022e447d8af0d20d77bc0ea41e0e1451012e1185e97e58f13409b6542bc19                                                                                                                                                                                                                                                                                                                                           0.0s
 => => naming to docker.io/library/stackoverflowtest

    You can see that in the example above, the step 2 and 3 have changed because bigFiles is also part of app, so no layer could be cached.

    However, if you only change the small files and not big files, the big file layer can be cached:

    $ echo "Change me" > app/smallFile
$ docker build -t stackoverflowtest .
[+] Building 0.9s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 36B                                                                                                                                                                                                                                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/python:latest                                                                                                                                                                                                                                                                                                                                                                       0.6s
 => [1/4] FROM docker.io/library/python:latest@sha256:e9c35537103a2801a30b15a77d4a56b35532c964489b125ec1ff24f3d5b53409                                                                                                                                                                                                                                                                                                                 0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => => transferring context: 148B                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => CACHED [2/4] COPY ./app/bigFiles .                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => [3/4] COPY ./app .                                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => [4/4] RUN echo 'Done'                                                                                                                                                                                                                                                                                                                                                                                                              0.1s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                                                                                                                                                0.0s
 => => writing image sha256:fdb687373689497d81bd6a14d4c30c6586b613b25463cdf5b387475700ccfb07                                                                                                                                                                                                                                                                                                                                           0.0s
 => => naming to docker.io/library/stackoverflowtest
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search