Question 80411 in Docker Question posted in
The official documentation can be found here.

Containerizing Python script with read and write access to local files not in project folder – Docker

I store some files in a Google Drive folder and have been running my scripts locally, so that the scripts read files from the GDrive folder and then save some outputs to the GDrive folder.

I now want containerize a script with the function do_stuff() so that it can access the google_drive_folder to both read data and write to it.

import os
from pickle import load, dump
import my_script

google_drive_folder = 'C:\Users\Me\Google Drive\Projects'

source_file_location = os.path.join(google_drive_folder, 'old_data.pickle')
with open(source_file_location, 'rb') as read:
    old_data = load(read)

new_data = my_script.do_stuff(old_data)

target_file_location = os.path.join(google_drive_folder, 'new_data.pickle')
with open(target_file_location, 'wb') as fp:
    dump(new_data, fp)

print(f'File saved to {target_file_location}')

The error message I get looks like this, even though when using Windows Explorer I can copy paste that exact path and find the file I’m looking for.

FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/Me/Google Drive/Projects/old_data.pickle'

All the scripts are in a folder called src. I’m trying to use docker-compose and I’ve tried storing different variables like the folder locations to a .env file without success.

So far my docker file looks like this

FROM python:3.10-slim

WORKDIR /code

COPY ./requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY ./src ./src

CMD ["python", "./src/my_script.py"]

docker-compose.yml looks like this

services:
  app:
    build: .
    container_name: my-scripts
    command: python ./src/my_script.py
    volumes:
      - .:/code
    env_file:
      - .env
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Alright I did some digging and figured out exactly what I needed to do. The below configurations make it so that data from Google drive is accessible from the container like this

    source_file_location = os.path.join('data', 'old_data.pickle')
with open(source_file_location, 'rb') as read:
    old_data = load(read)

    docker-compose.yml which creates a bind mount data, but not a folder visible in the file system of the container

    services:
  app:
    build: .
    container_name: my-scripts
    command: python ./src/my_script.py
    volumes:
      - .:/code
      - type: bind
        source: "C:/Users/Me/Google Drive/Projects"
        target: /code/data
    env_file:
      - .env

    Dockerfile just builds the image, all the logic related to bridging the host and container folders is in the docker-compose

    FROM python:3.10-slim
WORKDIR /code
COPY ./src ./src
COPY ./requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
CMD ["python", "./src/my_script.py"]

  2. 0

    Once you are on a container, you are no longer in your "normal OS".

    For example, python container is based on linux, as a result, there is no such a folder as C:whatever.

    What you have to do here is to use a volume that you will mount where your picke file will be.

    in container realm

    You will have a folder, let’s call it /data and in this folder you will have a old_data.pickle file.

    When you run your script in the container, it will have to access the /data folder, and will write the new_data.pickle inside it as well.

    in the host real

    You will still have a C:UsersMeGoogle DriveProjects folder. In this folder the old and new files will appear.

    make the link between two realms

    The link is done by the "volume" statement on you compose file, you will have : C:UsersMeGoogle DriveProjects:/data on your "volume" part.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search