Question posted in
The official documentation can be found here.

Python celery creates a lot of connections on redis

I am trying to make internet service using redis and celery.
My redis instance is tied to 10 simultaneous clients. When I am trying to run just one worker it initiates 6-9 connections for some reason (I see this on my redis admin panel). After some time it starts throwing ‘max clients reached’ error.
Is this expected behavior that worker is using so many connections? Should I switch to rabbitmq?

config = Configuration(None, EnvironmentType.beta, '../configuration.json')
 
beta_broker = Celery('src.beta_module', broker=config.celery_redis_url, include=['src.beta_module.notifications'])



 
 
def main():
    beta_broker.start(argv=['celery', 'worker', '-E', '--concurrency=1', '--loglevel=DEBUG'])
 
 
if __name__ == '__main__':
    main()

UPDATED

I tried to set up redis_max_connections, but it does not help

    beta_broker = Celery('src.beta_module', broker=config.celery_redis_url, include=['src.beta_module.notifications'])
    beta_broker.conf.redis_max_connections = 2

broker_pool_limit = 0
BROKER_TRANSPORT_OPTIONS = {
‘max_connections’: 20,
}

still creates 8 connections to redis

Is there any way to disable receiving results of the tasks? I don’t need this functionality anyway. Maybe it will help to reduce the number of connections?

UPDATED

After about 3 days of researching and reading numerous blogs and docs I decided to switch to rq

Tags:

2

Answers


  1. 0

    have you tried set max connections for redis connetion pool with celery? https://docs.celeryproject.org/en/stable/userguide/configuration.html#redis-max-connections

    redis_max_connections – Default: No limit.

    Login or Signup to reply.
  2. 0

    Try the following:

    beta_broker.conf["broker_transport_options"] = {'fanout_prefix': True,
                                                'fanout_patterns': True,
                                                'max_connections': 2,
                                                'socket_keepalive': True},
beta_broker.conf["broker_pool_limit"] = 2
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search