I’ve been trying to create a container to run in Dagster cloud on an ECS-hybrid deployment model. I’m able to actually push the container to Dagster but I continually get this error:
dagster_cloud.workspace.ecs.client.EcsServiceError: ECS service failed because task arn:aws:ecs:ap-northeast-1:*****:task/Dagster-Cloud-my-cluster-Cluster/1d151e6d40b44588a4ed4446a949d44a failed: CannotStartContainerError: ResourceInitializationError: failed to create new container runtime task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "dagster": executable file not found in $PATH: unknown
Task logs:
runc create failed: unable to start container process: exec: "dagster": executable file not found in $PATH
For more information about the failure, check the ECS console for logs for task arn:aws:ecs:ap-northeast-1:*****:task/Dagster-Cloud-my-cluster-Cluster/1d151e6d40b44588a4ed4446a949d44a in cluster Dagster-Cloud-my-cluster-Cluster.
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1304, in _reconcile
self._wait_for_new_server_ready(
File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 458, in _wait_for_new_server_ready
task_arn = self.client.wait_for_new_service(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 491, in wait_for_new_service
return self.check_service_has_running_task(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 607, in check_service_has_running_task
self._raise_failed_task(task, container_name, logger)
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 526, in _raise_failed_task
raise EcsServiceError(
I’m not sure why this is happening as I’m sure I’m installing dagster and dagster-cloud, as per the documentation. My Docker container looks like this:
###############################################################################
# Base container
###############################################################################
FROM python:3.11 AS base
# Define the environment variables necessary to work with poetry
ENV POETRY_VERSION=1.5.1
POETRY_HOME="/opt/poetry"
POETRY_VIRTUALENVS_IN_PROJECT=true
POETRY_NO_INTERACTION=1
# Add the poetry bin to our path
ENV PATH="$POETRY_HOME/bin:$PATH"
###############################################################################
# Poetry installer container
###############################################################################
FROM base AS installer
# Install poetry
RUN curl -sSL https://install.python-poetry.org | python3 -
###############################################################################
# Container that actually builds the application
###############################################################################
FROM base AS builder
# Copy the poetry files from the poetry installer to this container
COPY --from=installer $POETRY_HOME $POETRY_HOME
# Describe the environment variables necessary to install our dependencies
ENV PYTHONFAULTHANDLER=1
PYTHONUNBUFFERED=1
PYTHONHASHSEED=random
PIP_NO_CACHE_DIR=off
PIP_DISABLE_PIP_VERSION_CHECK=on
PIP_DEFAULT_TIMEOUT=100
# Set the working directory to one we can install to easily
WORKDIR /app
# Copy the poetry.lock and pyproject.toml files first to ensure that dependencies
# are only installed when they're updated
COPY poetry.lock pyproject.toml /app/
# Copy in our SSH key so we can retrieve the shared GitHub repo
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# Install our project dependencies
RUN --mount=type=ssh poetry install --only dagster --no-ansi --no-root
# Build the project
COPY . /app/
RUN poetry build
###############################################################################
# Create a runtime environment that's much smaller
###############################################################################
FROM python:3.11-alpine AS runtime
# Set the working directory to where Dagster will look for the application
WORKDIR /opt/dagster/app
# Copy the project wheel file
COPY --from=builder /app/dist/*.whl /
# Install the wheel file using pip and then install dagster and dagster-cloud
RUN pip install --no-cache-dir /*.whl
&& rm -rf /*.whl
For this package, my pyproject.toml
file lists my dependencies as follows:
[tool.poetry.group.dagster.dependencies]
dagster = "^1.5.9"
dagster-aws = "^0.21.9"
pendulum = "^2.1.2"
pandas = "^2.1.3"
openpyxl = "^3.1.2"
dagster-cloud = "^1.5.12"
So, both of these should be installed, but clearly they haven’t been. What am I doing wrong here?
2
Answers
This container had two issues:
To fix these issues, I used Poetry to construct a
requirements.txt
file and then used pip to install the requirements from it. As these included bothdagster
anddagster-cloud
, the package ended up with both necessary executables.You probably forgot to activate the virtual environment? In case it is located in /.venv , I suggest to add
at the end of the Dockerfile.