Question posted in
The official documentation can be found here.

Make docker env variables from an `.env` file available in build step (Dockerfile) & during run-time in container

Premises

Given a file oneSourceOfTruth.env:

FOO=42
... (many entries)

and a docker-compose.yml:

services:
  my-service:
    dockefile: ./Dockerfile
    env_file: oneSourceOfTruth.env

🏁 Objective

I’d like to have all variables from oneSourceOfTruth.env available in the Dockerfile during the build step via docker compose build as well as in the container during runtime (docker compose up). The variables are static in the sense that they are once set (before the build) and then never change again for a specific build and the containers that spawn from the build.

Unfortunately, it does not work with env_file option as it only passes the env variables to the containers. The env variables are not available during the build (e.g. with a RUN command inside the Dockerfile).

Sidenote: I don’t necessarily need to access the env variables directly inside the docker-compose.yml file itself, e.g. we don’t have something like the following in our docker-compose.yml file.

args:
   - FOO=$FOO

⭕ Constraint

The oneSourceOfTruth.env file is quite long. For scalability reasons and better code quality, I’d like to regard this file as the "one source of truth" where env variables are declared and set. That is, introducing (removing) an env variable should only imply I have to add (remove) one line in this file alone.

❎ Ways to solve the objective without respecting the constraint.

ARG FOO
ENV FOO $FOO

in your Dockerfile. Do this for every single variable. One user knows what I feel about this approach (see here):

This is not a practical solution whatsoever if you have a big environment file. This makes development tedious and unnecessarily annoying where if you can just read environment variables passed from the compose file. Hard-coding variables is never good, because if you would then want to add more environment variables you have to edit multiple files.

The solution proposed here source oneSourceOfTruth.env && docker-compose build did not work for me (env variables were not populated / were empty).

  • Pass the env varialbes in the environment section in the docker-compose.yml file as described here and here.

Other answers I’ve checked out:

Avoid XY problem

Maybe this is an XY problem. For the sake of completeness, here is why I’d like to get this problem solved 😉

Inside our Dockerfile, we use the following Rails command to precompile our assets:

RUN DB_ADAPTER=nulldb bundle exec rails assets:precompile

Unfortunately, this will spawn the entire Rails machinery (at least we can avoid connecting to the DB with the DB_ADAPTER=nulldb adapter from here). There existed an option initialize_on_precompile that one could set to false to avoid booting up Rails for this task, see here). However, this option was removed from the Rails codebase, see this commit.

This means we are forced to load env variables also in the precompile task. As stated, with the key env_file in our docker-compose.yml file, the env variables are only available in the containers and not during the build (where we need them for the precompile task). A workaround so far was to use ENV("Foo", nil) everywhere in our ruby on rails code such that the variable default to nil if not specified. This way everything works: during the precompile task, all env variables are nil but we don’t need them anyways for the mere task of precompiling our assets. During the production run-time, the env variable will then be available.

But with this approach, we silently ignore the case when we really forget to set an env variable. Then, also during production, the variable will be nil and we only recognize this by the effects it causes. Therefore, our solution so far is really just a workaround and should be fixed. If a variable is not available, it should raise a KeyError during the build step of the project and not just when a user complains that something in our app is not working. This can be achieved by using ENV("Foo") instead of ENV("Foo", nil), but now we get the key error during the precompile task since the env variables are not available in the Dockerfile (where we execute the precompile task). That’s exactly the problem 😉

Tags:

2

Answers


  1. 0

    You have the oneSourceOfTruth.env available on the machine, where the Docker build happens, right?

    Then you can do something like this:

    # ...
WORKDIR /app
COPY oneSourceOfTruth.env /app/oneSourceOfTruth.env
RUN source /app/oneSourceOfTruth.env
RUN DB_ADAPTER=nulldb bundle exec rails assets:precompile
RUN rm -f /app/oneSourceOfTruth.env
# ...

    Please bear in mind, that the final Docker image should not contain any secrets. Therefore I’d recommend deleting the oneSourceOfTruth.env from the image again file before finishing the build or do a multi-stage build where you do the build in the first step and copy the build result (without the oneSourceOfTruth.env) to the second build step.

    At runtime, pass the env_file in your Docker Compose config like you already did.

    Login or Signup to reply.
  2. 0

    In addition to not matching well with Docker’s ENV primitive, the other practical problem with this setup is that the environment gets reset at the end of each RUN command.

    If you only need the contents of the file once, then you can use the standard shell . command to read it in within the context of a single RUN command. (Some shells have a similar source command, but it is not part of the shell standard, and I’d avoid it in most cases.)

    WORKDIR /app
COPY ./ ./ # includes .env file
RUN . oneSourceOfTruth.env && 
    DB_ADAPTER=nulldb bundle exec rails assets:precompile
# RUN echo $Foo  # environment variables won't be set any more
# CMD echo $Foo  # nor in the eventual container

    You can work around the CMD problem by writing a shell script that reads in the environment file and then runs the command it’s passed. That script can be the image’s ENTRYPOINT.

    #!/bin/sh
. /app/oneSourceOfTruth.env
exec "$@"

    ENTRYPOINT ["/app/with-environment"]
CMD echo $Foo  # will be set now

    If you need the environment to be set on every RUN command, then you can override the Dockerfile SHELL to run this script too. This defines the interpreter that’s used for shell-syntax RUN and CMD commands (not JSON-array syntax), and you could inject a wrapper script here too.

    SHELL ["/app/with-environment", "/bin/sh", "-c"]
RUN echo $Foo
CMD echo $Foo
# CMD ["/bin/sh", "-c", "echo $Foo"] # will not have the environment

    The one caution with this last setup is that a docker run or Compose command: override won’t see the Dockerfile SHELL and the environment variables won’t be set there. You need the ENTRYPOINT wrapper for that.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search