I have 4 services running using the docker-compose
1) python-api
2) python-model
3)python-celery
4)redis-server
.
Flow:
1) python-api
gets hit via postman with images and some text as parameters on port 8000.
2) python-api
passes the image and data to python-model
on port 8001 for some ML predictions.
3) The modified image and response data in JSON format is then passed to python-celery
for triggering mails.
Error: python-celery
is able to grab hold of images and responses that are being sent by python-model
in step3. But it’s not able to read image currently
Error log:
========================
python-celery_1 | Received task: classify_crack.tasks.queue_task_v3[d71f976f-b2e7-4b29-9147-35996668de17]
python-celery_1 | == unique_file_index
python-celery_1 | AANJkaNIJSDHURHQEYRQ(*R
python-celery_1 | /python-model/server/classify_crack/inference/images/202003251237371/202003251237371_0.jpg 64
python-celery_1 | Task classify_crack.tasks.queue_task_v3[d71f976f-b2e7-4b29-9147-35996668de17] raised unexpected: AttributeError("'NoneType' object has no attribute 'shape'",)
python-celery_1 | Traceback (most recent call last):
python-celery_1 | File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
python-celery_1 | R = retval = fun(*args, **kwargs)
python-celery_1 | File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
python-celery_1 | return self.run(*args, **kwargs)
python-celery_1 | File "/python-api/classify_crack/tasks.py", line 690, in queue_task_v3
python-celery_1 | save_heat_map_v2(predictions, img_path, _dir, unique_file_index, original_image_index, i, grid_size=grid_size, metadata=metadata, StoredFileLinks=StoredFileLinks, row_stride=row_stride, col_stride=col_stride)
python-celery_1 | File "/python-api/classify_crack/tasks.py", line 128, in save_heat_map_v2
python-celery_1 | num_row_splits = int(np.ceil(img.shape[0]/row_stride))
python-celery_1 | AttributeError: 'NoneType' object has no attribute 'shape'
Line inside python-celery
code where I’m getting an error:
img = cv2.imread(img_path)
print(img_path)
print(img)
# print(img_path,grid_size)
splits = int(np.ceil(img.shape[0]/row_stride))
Here, img_path
is a valid path inside the container that is being printed. But I’m not able to read the image as img
returns None
. and the line splits
giving me the above error.
Reason why I’m getting an error:
I’m receiving this error because it is trying to access the folder path:
/python-model/server/classify_crack/inference/images/202003251237371/202003251237371.jpg
, but python-celery
is not able to access that folder with the name 202003251237371
.
Proof:
I tried using the command:
command: >
sh -c "ls '/python-model/server/classify_crack/inference/images' &&
inside the docker-compose of both python-model
and python-celery
services, I get the following outcome while I run all the containers again:
python-model_1 | 201801151543500
python-model_1 | 201801151543500.jpg
python-model_1 | IMG_20190307_184100
python-model_1 | IMG_20190307_184100.jpg
python-model_1 | extracted_input_0_0 (15)
python-model_1 | extracted_input_0_0 (15).jpg
python-model_1 | extracted_input_0_0 (16)
python-model_1 | extracted_input_0_0 (16).jpg
python-model_1 | extracted_input_0_0 (18)
python-model_1 | extracted_input_0_0 (18).jpg
python-model_1 | extracted_input_0_0 (19)
python-model_1 | extracted_input_0_0 (19).jpg
python-model_1 | extracted_input_0_0 (9)
python-model_1 | extracted_input_0_0 (9).jpg
python-model_1 | file
python-model_1 | image (2)
python-model_1 | image (2).png
python-model_1 | 202003251237371
python-model_1 | 202003251237371.jpg
python-model_1 | image_X
python-model_1 | image_X.png
python-model_1 | original
python-model_1 | original.jpg
python-model_1 | original_image_0
python-model_1 | original_image_0.jpg
python-celery_1 | 20013V_Y.JPG
python-celery_1 | extracted_input_0_0 (15).jpg
python-celery_1 | extracted_input_0_0 (16).jpg
python-celery_1 | extracted_input_0_0 (18).jpg
python-celery_1 | extracted_input_0_0 (19).jpg
python-celery_1 | extracted_input_0_0 (4).jpg
python-celery_1 | 202003251237371.jpg
Now clearly, python-celery
cannot display folder 202003251237371
with the image name 202003251237371.jpg
, which I could see in python-model
.
How to tackle this scenario and allow python-celery
to access such image folders?
docker-compose
version: "3"
networks:
app-tier:
driver: bridge
volumes:
app-volume: {}
services:
python-api-celery: &python-api-celery
build:
context: /Users/AjayB/Desktop/python-api/
networks:
- app-tier
volumes:
- app-volume:/python-model/server/classify_crack/:rw
environment:
- PYTHON_API_ENV=development
command: >
sh -c "python manage.py makemigrations &&
python manage.py migrate"
python-api: &python-api
<<: *python-api-celery
ports:
- "8000:8000"
command: >
sh -c "python manage.py runserver 0.0.0.0:8000"
python-celery: &python-celery
<<: *python-api-celery
depends_on:
- redis
links:
- python-model
command: >
sh -c "ls '/python-model/server/classify_crack/inference/images' &&
celery -A server worker -l info"
redis:
image: redis:5.0.8-alpine
hostname: redis
networks:
- app-tier
expose:
- "6379"
volumes:
- app-volume:/python-model/server/classify_crack/:rw
ports:
- "6379:6379"
command: ["redis-server"]
python-model: &python-model
build:
context: /Users/AjayB/Desktop/Python/python/
ports:
- "8001:8001"
networks:
- app-tier
environment:
- PYTHON_API_ENV=development
volumes:
- app-volume
depends_on:
- python-api
command: >
sh -c "ls '/python-model/server/classify_crack/inference/images' &&
cd /python-model/server/ &&
python manage.py migrate &&
python manage.py runserver 0.0.0.0:8001"
Instance of containers:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7374a7b0b051 integrated_python-celery "sh -c 'celery -A se…" 13 minutes ago Up 5 seconds 8000/tcp integrated_python-celery_1
8eb9a754996a integrated_python-model "sh -c 'cd /python-m…" 20 minutes ago Up 5 seconds 0.0.0.0:8001->8001/tcp integrated_python-model_1
b268b7bd1ac4 integrated_python-api-celery "sh -c 'python manag…" 20 minutes ago Up 6 seconds 8000/tcp integrated_python-api-celery_1
869bb5fc21b2 integrated_python-api "sh -c 'python manag…" 20 minutes ago Up 6 seconds 0.0.0.0:8000->8000/tcp integrated_python-api_1
c85a1becea34 redis:5.0.8-alpine "docker-entrypoint.s…" About an hour ago Up 6 seconds 0.0.0.0:6379->6379/tcp integrated_redis_1
3
Answers
The error got solved at last:
It worked with including following lines in python-api's
views.py
at the start:also, modified the volume name in
docker-compose
, just to avoid some confusion. Can't believe it took almost 1 week to get this solved.docker-compose:
In my opinion, you should follow an in-depth tutorial about how to use celery with python API with Docker and Kubernetes 🙂
From the OpenCV-Python doc: `
Which seems exactly what happens here. Maybe your image path seems right but is actually wrong, possible causes:
/python-model/server/classify_crack/inference/images/202003251237371/202003251237371.jpg
does not allow the user runningpython-celery
to read itYour Compose volume definition looks fine otherwise.