Question posted in
The official documentation can be found here.

Issue with Python App in Docker Compose – ModuleNotFoundError:

I have a monorepo (turbo) containing several apps, one of which is a Python app. I’ve set up a Dockerfile for the app, but I’m encountering an issue during the build process.

Here’s my Dockerfile:

FROM node:19.9.0-alpine as base

# adding apk deps to avoid node-gyp related errors and some other stuff. adds turborepo globally
RUN apk add -f --update --no-cache --virtual .gyp nano bash libc6-compat g++ 
    && yarn global add turbo 
    && apk del .gyp

RUN apk --no-cache add make python3 py3-pip

#############################################
FROM base AS pruned
WORKDIR /app
ARG APP

COPY . .

# see https://turbo.build/repo/docs/reference/command-line-reference#turbo-prune---scopetarget
RUN turbo prune --scope=sheet_sync --docker

#############################################
FROM base AS installer
WORKDIR /app
ARG APP

COPY --from=pruned /app/out/json/ .
COPY --from=pruned /app/out/yarn.lock /app/yarn.lock

# Forces the layer to recreate if the app's package.json changes
COPY services/sheet-sync/package.json /app/services/sheet-sync/package.json

# see https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#run---mounttypecache
RUN 
    --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,sharing=locked 
    yarn --prefer-offline --frozen-lockfile

COPY --from=pruned /app/out/full/ .
COPY turbo.json turbo.json

# For example: `--filter=frontend^...` means all of frontend's dependencies will be built, but not the frontend app itself (which we don't need to do for dev environment)
RUN turbo run build --no-cache --filter=sheet_sync^...

# re-running yarn ensures that dependencies between workspaces are linked correctly
RUN 
    --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,sharing=locked 
    yarn --prefer-offline --frozen-lockfile

#############################################
FROM base AS runner
WORKDIR /app

COPY --from=installer /app .

CMD yarn workspace sheet_sync start

And here’s the relevant part of my package.json:

{
    "name": "sheet_sync",
    "version": "0.0.1",
    "scripts": {
      "start": "make run",
      "debug": "make debug",
      "build": "make build"
    }
}

And my Makefile:

.PHONY: test
test:
    pytest -svv -s
    
.PHONY: build
build:
    python3 -m venv googleSheetENV
    source googleSheetENV/bin/activate && python3 -m pip install -r requirements.txt

.PHONY: run
run:
    python3 main.py

The issue arises during the Docker build process, and the error message is as follows:

yarn workspace v1.22.19
yarn run v1.22.19
$ make run
python3 main.py
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.
Traceback (most recent call last):
  File "/app/services/sheet-sync/main.py", line 1, in <module>
    from src.reads.supplier_info import build as build_supliers
  File "/app/services/sheet-sync/src/reads/supplier_info.py", line 1, in <module>
    from src.reads.read_data_from_sheet import read_data_from_sheet
  File "/app/services/sheet-sync/src/reads/read_data_from_sheet.py", line 1, in <module>
    from src.config.google_config import sheets
  File "/app/services/sheet-sync/src/config/google_config.py", line 2, in <module>
    from dotenv import load_dotenv
ModuleNotFoundError: No module named 'dotenv'
make: *** [Makefile:12: run] Error 1
error Command failed with exit code 2.
error Command failed.
Exit code: 2
Command: /usr/local/bin/node
Arguments: /opt/yarn-v1.22.19/lib/cli.js start
Directory: /app/services/sheet-sync
Output:

requirements.txt

absl-py==1.4.0
annotated-types==0.6.0
anyio==3.7.1
aws-lambda-powertools==2.19.0
beautifulsoup4==4.12.2
black==23.3.0
blinker==1.6.3
build==0.10.0
cachetools==5.3.1
certifi==2023.7.22
charset-normalizer==3.3.0
click==8.1.7
colorama==0.4.6
dnspython==2.4.2
fastapi==0.103.2
filelock==3.12.4
fire==0.5.0
flake8==6.0.0
Flask==3.0.0
Flask-Cors==3.0.10
gdown==4.7.1
google-api-core==2.12.0
google-api-python-client==2.101.0
google-auth==2.23.2
google-auth-httplib2==0.1.1
google-auth-oauthlib==1.1.0
googleapis-common-protos==1.60.0
grpc-google-iam-v1==0.12.6
grpcio==1.56.2
grpcio-status==1.56.2
gspread==5.11.3
gspread-formatting==1.1.2
gunicorn==21.2.0
h11==0.14.0
httplib2==0.22.0
idna==3.4
imageio==2.31.5
iniconfig==2.0.0
isort==5.12.0
itsdangerous==2.1.2
Jinja2==3.1.2
lazy_loader==0.3
markdown-it-py==3.0.0
MarkupSafe==2.1.3
mccabe==0.7.0
mdurl==0.1.2
mypy==1.4.1
mypy-extensions==1.0.0
networkx==3.1
numpy==1.26.0
oauthlib==3.2.2
opencv-python==4.8.1.78
opencv-python-headless==4.8.1.78
ortools==9.6.2534
packaging==23.2
pathspec==0.11.1
Pillow==10.0.1
platformdirs==3.8.1
pluggy==1.3.0
proto-plus==1.22.3
protobuf==4.24.3
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycodestyle==2.10.0
pydantic==2.4.2
pydantic_core==2.10.1
pyflakes==3.0.1
Pygments==2.16.1
pymongo==4.6.1
pyparsing==3.1.1
pyproject_hooks==1.0.0
PySocks==1.7.1
pytest==7.4.2
python-dotenv==0.19.0
pytz==2023.3.post1
PyYAML==6.0.1
requests==2.31.0
requests-oauthlib==1.3.1
rich==13.6.0
rsa==4.9
scikit-image==0.22.0
scipy==1.11.3
six==1.16.0
sniffio==1.3.0
soupsieve==2.5
starlette==0.27.0
tabulate==0.9.0
termcolor==2.3.0
tifffile==2023.9.26
tqdm==4.66.1
typing_extensions==4.8.0
uritemplate==4.1.1
urllib3==2.0.6
uvicorn==0.23.2
Werkzeug==3.0.0

I suspect the issue is related to the Docker build process or potentially incorrect installation of necessary libraries, particularly the absence of the dotenv module during the build, even though it’s present in my Python code.

I’ve already tried including python-dotenv in my requirements.txt, but the problem persists.

Tags:

2

Answers


  1. 0

    dotenv isn’t in your requirements.txt

    Login or Signup to reply.
  2. 0

    In your Makefile, the build target is using a virtual environment, but the run target isn’t. That’s leading to the error you see.

    If you want to keep your existing system, you should make sure to consistently use the virtual environment everywhere. I’d recommend using Make’s standard dependency system to ensure things like "the virtual environment exists" and "packages are installed": it is a build-coordination system and not just a script runner.

    # Location of the virtual environment
VIRTUAL_ENV := googleSheetENV
# Python executable inside the virtual environment
PYTHON := $(VIRTUAL_ENV)/bin/python3
# Test runner; something installed by requirements.txt
PYTEST := $(VIRTUAL_ENV)/bin/pytest

# Artificial build targets
.PHONY: dependencies build test run
all: run
dependencies: $(PYTEST)
build: dependencies

# Create the virtual environment using the system Python
$(PYTHON):
        python3 -m venv $(VIRTUAL_ENV)

# Install dependencies, hinging on an arbitrary binary known to exist
$(PYTEST): requirements.txt $(PYTHON)
        $(PYTHON) -m pip install $<

# Run tests
test: $(PYTEST)
        $(PYTEST) -svv -s

# Run the application
run: $(PYTEST)
        $(PYTHON) main.py

    Walking through this backwards: at the end, make run runs $(PYTHON)/main.py, which uses the virtual-environment copy of Python. It depends on $(PYTEST), which runs the pip install step, and that in turn depends on $(PYTHON), which creates the virtual environment initially. The Make dependencies mean that, if you make run again (in a non-container environment), it won’t repeat the initial installation steps.

    If this seems complicated for "run a Python application in a container", it is. Your setup is using a Javascript build manager to invoke a C build manager to invoke the Python build system. Further, a Docker image is already an isolated environment, and the image’s "system" Python is isolated from any other Pythons that may exist; in most cases you don’t need a virtual environment in Docker.

    Your setup presumably has other Javascript parts, but for just this application I’d use a routine Python Dockerfile

    FROM python:3.12
WORKDIR /app
COPY requirements.txt ./
RUN pip install -r requirements.txt
COPY ./ ./
CMD ["/app/main.py"]

    and discard the Makefile and Javascript package.json.

    You’d need to run this in a separate container from your Javascript components, and use a separate image. That’s probably fine, though, and you need to run multiple containers already since a container only runs one program.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search