Question posted in
The official documentation can be found here.

Redis – How to monitor Celery task completion with Prometheus

I am trying to use Prometheus for monitoring Celery tasks for which I am relatively new and I have a problem with incrementing a counter. It’s just not incrementing if I am trying to do it inside of Celery.task

E.g.

from celery import Celery
from prometheus_client import Counter

app = Celery('tasks', broker='redis://localhost')
TASKS = Counter('tasks', 'Count of tasks')


@app.task
def add(x, y):
    TASKS.inc(1)
    return x + y

When I visit an endpoint to see which metrics are exposed I can see tasks_total, but its value doesn’t change no matter how much add tasks have been executed.
However, when I am trying to increment the same counter from regular function, it works.

E.g. 

def dummy_add(x, y):
    TASKS.inc()
    return x + y

Could you please explain to me what I am doing wrong?

Tags:

2

Answers


  1. 0

    By default, Celery uses process pool for workers. This doesn’t go well with Prometheus Python client and you have to use its multiprocessing mode.

    You might rather want to use one of existing Celery exporters (such as this one) which take a different approach. They just start its own process and listen on events from workers thus overcoming the drawbacks mentioned above.

    Login or Signup to reply.
  2. 0

    I tried the existing Celery exporter (OvalMoney/celery-exporter) and found some of the metrics to be missing or invalid.

    I’ve written an exporter for the purpose of monitoring task status here – https://github.com/danihodovic/celery-exporter.

    enter image description here

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search