Question 121358 in Redis Question posted in
The official documentation can be found here.

Redis – How to run parallel tasking with celery django?

I am looking to run tasks in parallel with django celery.

Let’s say the following task:

@shared_task(bind=True)
def loop_task(self):
    for i in range(10):
        time.sleep(1)
        print(i)
    return "done"

Each time a view is loaded then this task must be executed :

def view(request):
    loop_task.delay()

My problem is that I want to run this task multiple times without a queue system in parallel mode. Each time a user goes to a view, there should be no queue to wait for a previous task to finish

Here is the celery command I use :

celery -A toolbox.celery worker --pool=solo -l info -n my_worker1


-------------- celery@my_worker1 v5.2.7 (dawn-chorus)
--- ***** -----
-- ******* ---- Windows-10-10.0.22000-SP0 2022-08-01 10:22:52
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         toolbox:0x1fefe7286a0
- ** ---------- .> transport:   redis://127.0.0.1:6379//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 8 (solo)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

I have already tried the solutions found here but none of them seem to do what I ask StackOverflow : Executing two tasks at the same time with Celery

I should have the following output:

0,1,2,...,9

If two users load the same page at the same time then we should have the previous output appearing twice

Result :

0,0,1,1,2,2,...,9,9
Tags:

2

Answers


  1. 0

    Maybe need to start multi workers, but this does not guarantee that all tasks can be performed in parallel.

    Will still has task in doesn’t receive in queue. It depends on the number of workers and the speed of execution.

    And if same result, you can set it in cache.

    Login or Signup to reply.
  2. 0

    I think it’s very simple to solve, but you need to test it.

    Basically, you need to run task in async mode – for example, when you are trying to run task that send mass sms to multiple users, you do it in this way:

    send_mass_sms.apply_async(
    [
        phone_numbers,
        instance.body,
        instance.id,
    ],
    eta=instance.when,
)

    Your code needs to be fixed this way:

    def view(request):
    loop_task.apply_async()

    If you need to update data on website, you can store data in models and call ajax multiple times or implement logic via websockets, but this is topic for another question 🙂

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search