skip to Main Content

I am building a lead generation portal that can be accessed online. Please don’t mind the verbosity of the code, I’m doing a lot of debugging right now.

My Celery worker inconsistently picks up tasks assigned to it, and I’m not sure why.

The weird thing about this, is that sometimes it works 100% perfect: there never are any explicit errors in the terminal.

I am currently in DEBUG = TRUE and REDIS as a broker!

celery start worker terminal command and response

celery -A mysite worker -l info --pool=solo
 -------------- celery@DESKTOP-OG8ENRQ v5.0.2 (singularity)
--- ***** -----
-- ******* ---- Windows-10-10.0.19041-SP0 2020-11-09 00:36:13
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         mysite:0x41ba490
- ** ---------- .> transport:   redis://localhost:6379//
- ** ---------- .> results:     redis://localhost:6379/
- *** --- * --- .> concurrency: 12 (solo)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
[tasks]
  . mysite.celery.debug_task
  . submit

[2020-11-09 00:36:13,899: INFO/MainProcess] Connected to redis://localhost:6379//
[2020-11-09 00:36:14,939: WARNING/MainProcess] c:userscoolepycharmprojectslead_django_retryvenvlibsite-packagesceleryappcontrol.py:48: DuplicateNodenameWarning: Received multiple replies from node name: celery@DESKTOP-OG8ENRQ.
Please make sure you give each node a unique nodename using
the celery worker `-n` option.
  warnings.warn(DuplicateNodenameWarning(

[2020-11-09 00:36:14,939: INFO/MainProcess] mingle: all alone
[2020-11-09 00:36:14,947: INFO/MainProcess] celery@DESKTOP-OG8ENRQ ready.

views.py

class LeadInputView(FormView):
template_name = 'lead_main.html'
form_class = LeadInput

def form_valid(self, form):
    print("I'm at views")
    form.submit()
    print(form.submit)
    return HttpResponseRedirect('./success/')

tasks.py

@task(name="submit")
def start_task(city, category, email):
    print("I'm at tasks!")
    print(city, category, email)
    """sends an email when feedback form is filled successfully"""
    logger.info("Submitted")
    return start(city, category, email)

forms.py

class LeadInput(forms.Form):
    city = forms.CharField(max_length=50)
    category = forms.CharField(max_length=50)
    email = forms.EmailField()

    def submit(self):
        print("I'm at forms!")
        x = (start_task.delay(self.cleaned_data['city'], self.cleaned_data['category'], self.cleaned_data['email']))
        return x

celery.py

from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
app = Celery('mysite')

app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

settings.py

BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'

The runserver terminal will look something like this:

I'm at views
I'm at forms!
<bound method LeadInput.submit of <LeadInput bound=True, valid=True, fields=(city;category;email)>>

But the worker doesn’t say that it picked up anything, just that "celery@DESKTOP-OG8ENRQ ready." Except, when it does work… for some reason? I’m at a loss!

2

Answers


  1. Chosen as BEST ANSWER

    Hello to whoever sees this. It turns out, that this is a bug with celery (or maybe redis?)... apparently many windows users run into this. https://github.com/celery/celery/issues/3759

    Turns out, the answer is to make -P solo when starting worker. I'm not sure why this is the case... but that solved it!

    Thank you Naqib for your help! You put me down the right rabbit hole to a solution.


  2. By default, celery will use the hostname as worker name if your willing to use multiple workers in the same host then specify -n option.

    celery -A mysite worker -l info --pool=solo -n worker2@%h
    

    Your code works fine but the task is passed to the first worker, see
    DuplicateNodenameWarning with no obvious reason #2938

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search