Question posted in
The official documentation can be found here.

Redis – Sharing a single connection to mongo db between all celery workers

I have a series of celery workers which carry out tasks, stored using REDIS; in particular I have three categories of workers which respectively carry out three categories of tasks. The tasks belonging to each of these categories require access to a mongo DB. For reasons of efficiency I would like to use a single connection to the DB to be used by all workers. So far I’ve tried both passing as argument to "send_task" the connection in the following way:

myclient = MongoClient('localhost:27017')
celeryWorker.send_task('tasks.beampolyline', myclient)

but clearly it returns as error that the object is not JSON serializable; both trying to share the myclient object among all the workers but with poor results. Any idea ? I feel I am very close to the solution but I am stuck on both approaches I am trying to use.

Tags:

2

Answers


  1. 0

    One solution is to define module level connection. Suppose worker.py as your worker module:

    from mongodb import MongoClient

shared_connection = MongoClient()

@app.task(...)
def task(self, a, b):
    ...  # use shared_connection inside task
    return

    In this solution, connection is shared between threads/processes of this worker.

    Login or Signup to reply.
  2. 0

    Can’t you just do something similar?:

    import pymongo
from pymongo import MongoClient
from flask import *

client = pymongo.MongoClient("yourcredentials")
db = client['ALLWORKER']
worker_1 = db['worker1']
worker_2 = db['worker_2']
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search