So I am currently working on a DevOps task for a Python repository.
I am tasked with making the repository work properly with our Docker environment.
My current issue is that when running pip3 install
, it installs everything in various folders that have nothing to do with the current repository path.
This is not ideal when working with docker-compose
for a local development environment.
What we want to achieve is to install pip
dependencies once on the host machine and then mount them as a dynamic volume inside the docker container. The reason is simple, we want to be able to destroy the container as many times as we want without having to download everything again. Because this is an AI repository, you can imagine that the pip
dependencies are heavy and take a lot of time to download. We currently have about 5 to 10GB of dependencies to download depending on the branch and current code progress.
Ideally, I’d like pip
to behave like Node.JS tools such as yarn
or npm
where it builds a local folder (like node_modules
) that I can easily mount inside the container.
I have looked through the pip
documentation but I simply can’t find a way to do this at all.
Is there a way to constrain pip
and python
to install dependencies inside a specific folder and to only use those dependencies?
2
Answers
Have you tried using virtual environments:
ps. simplist way to setup virtual environment is using (https://docs.python.org/3/library/venv.html) but there are other solutions to do that too.
Docker Approach
You can just split up your dependencies into a separate docker image which serves as the basis for your current image. This way you only need to build base-image once and then only when the dependencies change.
Base Image Example
Build with
docker build -t myImage .
Actual Image
Python Approach
In general pip provides the
--target
option to install the packages into a specific directory instead of thesite-packages
folder see documentation.However, the use of a virtual environment is recommended to separate the dependencies from the host into an environment, which can then be mounted into a docker-container.
Then you can just mount
/path/myVenv
into the container andsource
again to access it.