I am wondering if it is possible to swap out a layer of a container image for another. Here is my scenario:
I have a docker file that does the following:
- It pulls the .net core 3.1 debian slim (buster) image
- Builds the source code of my application and adds it as a layer on top of the .net core image
When a new version of the .net core 3.1 runtime image comes out, I would like to make a new image that has that new version, but has the same application layer on top of it.
I do NOT want to have to find the exact version of the code I used to build the application and re-build.
The idea is to replicate upgrading the machine and runtime, but not have any alteration to the application (less to test with the upgrade).
Is there a docker command that I can use to swap out a layer of an image?
4
Answers
Is it possible? Yes. But it’s rarely done because of how error prone it is. For example, if the previous build would have created a library, but the library already exists in the original base image, that library won’t be included in the first build. If the base image removes that library, then the resulting merged image will be missing that library since it’s not in the new base layers and isn’t in the old application layer.
There are a few ways I can think of to do this.
Option 1: If you know the specific files in one image, then use the
COPY --from
syntax to copy those files between images. This is the least error prone method, but requires that you know every file you want to include. The resulting Dockerfile looks like:Option 2: you can export the images and create your own new image by combining the layers between the two. For this, there’s
docker save
anddocker load
. That would look like:Option 3: this could be done directly to the registry with API calls. You would pull the old manifest, adjust it with the new layers, and then push any new layers and then the new manifest. This would require a fair bit of coding (see regclient/regclient for how I’ve implemented some of these API’s in Go).
Option 4: I know I’ve seen a tool that does this in a specific scenario but the name of it escapes me. I believe it required that you use their base images that were curated to reduce the risk of incompatibilities between versions, and limited what base images could be swapped, so I don’t think it was a general purpose tool.
Note that option 2 and 3 both require either manual steps or for you to write some code if you want to automate it. Because of how error prone this is (as described above) I don’t think you’ll find anyone maintaining and supporting a tool to implement it. The vast majority rebuild from a Dockerfile using a CI tool.
This answer is broken into two distinct sections:
The first section can be skipped if potentially adverse side effects are not a concern.
Part I: Build From Sources Instead
No, there is probably not a
docker
command for this – and likely never will, due to a whole slough of technical issues. Even by design, this is not meant to be trivially possible; images and layers are meant to be immutable after being built. It is strongly recommended that the application image is instead rebuilt from the original sources, but with a new base image by modifying theFROM
command.There are numerous consistency issues that make this idea ill-advised, of which a few are listed below:
Certain
Dockerfile
commands do not actually create a layer, they update the final layer’s internal manifest and the image’s manifest. Commands that stateRemoving intermediate container XXXXXXXXXXXX
are actually updating these aforementioned manifests, and not creating new layers. This requires correctly updating the only relevant changes when swapping from the old base image to the new base image; e.g. reconciling changes fromENV
/LABEL
/MAINTAINER
/EXPOSE
/CMD
/ENTRYPOINT
commands.ENV
commands that alter the application image’s configuration from variables inherited in the previous base image may not be updated correctly. For example, in the application image, there might be the following commandENV
command:If the application image’s old base image layers are swapped out, and the new base image contains a different
${PATH}
variable, it is ambiguous how to reconcile the differences without a manual decision by a developer.If
apt
/apt-get
/apk
/yum
is used to install Linux packages, these packages are installed in the application image as subsequent layers. The installed packages are not guaranteed to be compatible with the new base image if the underlying OS or executables change in the new base image layers.Part II: A Proof-of-concept Implemented CLI Tool
"Upgrading" an image’s base is technically possible by doing direct manipulation on the image archive of the already-built Docker application image. However, I cannot reiterate this enough – you should not even be attempting to edit the bottom layers of an existing image. Seriously – stop it, get some help. I am 90% sure this is a war crime.
For the sake of humoring this question thoroughly though, I have developed a proof-of-concept CLI tool written in Java over on my GitHub project concision/docker-base-image-swapper that is designed to swap base images (but not arbitrary layers). The design choices I made to resolve various consistency issues are a "best guess" on what action should be taken.
Included is a demonstration for swapping the base image for an already-built Java application image from JDK8 to JDK11, implemented in the
demo/demo.sh
script. All core code is ran in isolated Docker contains, so only Bash and Docker are necessary dependencies on the host to run this demonstration. The demo application image is only built once on JDK 8, but is run twice – once on the original JDK 8 image, and another time on a swapped-base JDK 11 image.If you experience some technical difficulty with the tool, I may potentially be able to fix the issue. This project was quickly hacked together and has pretty poor code quality; furthermore, it likely suffers from various unaccounted edge cases. I might thoroughly rewrite this in Rust within the next month or two with a focus on maintainability and handling all edge cases.
The Concept
There are three images relevant in this process, all of which are already-built (i.e. no original sources are needed):
FROM
command in the originalDockerfile
.FROM old-base-image
layers.By knowing which layers and configurations of the application image are inherited from the old base image, they can be replaced with the layers and configurations from the new base image. All image layers and manifests can be obtained as a tar archive by using the
docker save
command. With an archive(s) of all relevant three images, a tool can analyze the differences betweenWarnings on Alternatives
Beware of doing simply a
COPY --from=...
from the old application image, as the original application’s image configuration (through commands such asCMD
,ENTRYPOINT
,ENV
,EXPOSE
,LABEL
,USER
,VOLUME
,WORKDIR
) will not be properly replicated.I have developed a small utility script using Python which lets you append a tarball to an existing container image in a container registry (without having to pull existing image data), available at https://github.com/malthe/appendlayer.
The following illustrates how it works:
The script supports any registry that implements the OCI Distribution Spec.
Two tools that provide this type of functionality that I have found are listed below.
NOTE: You do have to watch that you don’t have items that need updates in higher layers. And that higher layers do NOT have updates in them that would conflict with the new base. Example using image below the base is a TOMCAT container. If layer 4 of the original container did a update tomcat package that would overshadow the update of same files on the NEW BASE. Which could result in a non functional system. So as always your mileage may vary.