Question posted in
The official documentation can be found here.

Redis – Which way is the best for running background processes?

On the server-side: I need a way to execute some tasks in the background, frequently and start it at a specific time.
My programming language is Python for the back-end(Sanic Framework), VueJs for the front-end, MongoDB as main DB and the Redis for caching.
Also, I’m using a Docker container(docker-compose).
Also, I worked before with the Celery but I want to know what is the best solution for production that guarantees it’s stable and reliable.

On the client-side: For the mentioned question, I need to run it on the server-side, sometimes I need to run a job scheduler on clients, embedded devices such as Raspberry Pi that could run Python or JavaScript.

So, What are your solutions for these use cases?

Tags:

3

Answers


  1. 0

    Starting a new process in the background in python is as simple as calling os.fork(). For a comprehensive example, see https://python-course.eu/forking.php

    EDIT:

    For a fully featured solution, I’d recommend forking a background process as described above, and then using a library like https://github.com/dbader/schedule to execute jobs at scheduled intervals in that background process.

    Login or Signup to reply.
  2. 0

    In production we have both long and short-running tasks and in total our Celery cluster executes up to 6M tasks per day, so naturally I would recommend Celery. It is made for this purpose and if you are a Python developer you have another reason to pick Celery. Finally, Celery is the only Python task queue system known to me that has HA scheduler (https://github.com/mixkorshun/celery-beatx and https://github.com/sibson/redbeat).

    There are two other (Python) projects that should be mentioned as alternatives to Celery – Huey (https://github.com/coleifer/huey) and Apache Airflow (https://github.com/apache/airflow).

    Login or Signup to reply.
  3. 0

    I’m one of the core devs for Sanic. I would agree with the other answers that Celery is a great option. For anyone in need of a more light weight solution, I have a post about an alternative approach only inside Sanic: https://community.sanicframework.org/t/how-to-use-asyncio-queues-in-sanic/166/4

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search