Question 60948 in Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Trying to Run Celery Tasks on Remote Server – Ubuntu

I have been scouring the Internet for some sort of guide for setting up django, celery, and redis so the celery task runs on a remote server, and I cannot seem to find out how to do this.

I have three machines – tsunami (development), trout (production), and t-rex (Nvidia card). I am running some image manipulation algorithms using celery and redis. I have the same django/celery code base on all three computers. When I run the algorithms on tsunami or trout without CUDA support, it takes hours to complete. When I run the same algorithms on t-rex with CUDA support, they complete in minutes. What I would like to do is offload the celery tasks from trout and tsunami to t-rex, and return the results to the either trout or tsunami. By results, I mean a file to save on the local file system or data added to the local mysql (8.0.32) database. Is this even possible?

Can someone point me to a guide on how to set this up? I am running django 4.1.7, celery 5.2.7 and redis server 6.0.16. Trout and tsunami are running Ubuntu 20.04, and t-rex is Ubuntu 22.04. All are running Python 3.8.

I am concerned that when I install dlib on trout and tsunami, it is compiled without CUDA support because there are no Nvidia cards. All machines are running dlib 19.24.0, but only t-rex has it compiled with CUDA support. Is this a show stopper for my plan to use t-rex as a remote CUDA processor for the other machines?

Thanks!

Tags:

2

Answers


  1. 0

    You need setup different task queues and routes for your servers.

    Check out the celery docs on how to setup task routing

    To specify routes define task_routes. Let’s say you have your task that runs image manipulation defined under the module feed.tasks, use something like the following in your celery configuration to ensure that those tasks are only sent to the feeds queue:

    app.conf.task_routes = {'feed.tasks.*': {'queue': 'feeds'}}

    To specify what queues to consume from, modify your celery worker script and give a comma separated list of queues using the -Q option:

    $ celery -A proj worker -l INFO -Q foo,bar,baz,feeds

    By default celery worker will read from all queues. You can exclude queues on tsunami and trout, using the -X option

    $ celery -A proj worker -l INFO -X feeds
    Login or Signup to reply.
  2. 0

    Here servers can be docker images (docker-compose) or containers or pods (on k8s).
    Common tasks on all servers

    1. copy the project
    2. pip install -r requirements.txt

    server A: run django server as python manage.py runserver (if using uvicorn or gunicorn, then run the appropriate app server accordingly).

    server B: celery -A proj_name worker

    server C: celery -A proj_name worker -l INFO -Q foo,bar # logging level is Info and run tasks on queue foo and bar.

    server D: celery -A proj_name worker -l INFO -Q email,url # logging level is Info and runs tasks on queue email and url.

    Btw the location to start the django server and celery worker is the same, that is the project’s root directory, that is the directory where the manage.py is located.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search