Twitter API Question posted in
The official Twitter API documentation can be found here.

Scaling slightly long running Consumer work – RabbitMQ – Twitter API

I’m architecting RabbitMQ into our solution, and I’m curious on how to handle processing and acknowledging messages efficiently, while still performing “real” work in the consumer code that can range from 5-10 seconds. (More work than the samples ever delve into).

enter image description here

Above is an example of what I’m aiming to process. A message in my twitter.tweet_cmd_q queue that has all the parameters needed in the message body, for the consumer to make the actual Twitter API request, and save those results to the DB.

However, I’m running into two problems here:

  1. I’ll be processing thousands of records a minute – I can’t possibly take 5-10 seconds before acknowledging that message. Is it “normal” for the consumer code to process all the needed work before acknowledging the message? (i.e. I could see ack’ing the message, and throwing the actual work to be done in another thread for processing. – though this would then require it’s own form of “thread” management so that the system does undergo too much load).

  2. Would RPC calls benefit me at all here, in this situation that involves querying the data and saving it to the DB?

  3. Is the best way to deal with this scalability, just to create more worker instances for round robin handling?

Tags:

2

Answers


  1. 0

    1. Yes, it is normal for the consumer code to do everything it needs with the message before ack’ing it. That’s how you manage the message visibility to other workers.

    2. No, keep it simple, just do what you need to do in the worker.

    3. Sort of, it’s not really round robin, just create more workers, and have them subscribe to the queue. Each worker will poll the queue, find messages, and execute on them.

    Login or Signup to reply.
  2. 0

    The RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

    Rob’s answer is great, I just wanted to add to this:

    Is it “normal” for the consumer code to process all the needed work before acknowledging the message?

    If you acknowledge the message before processing it, and your consumer crashes or otherwise does not complete its task, that message is lost. That is the main reason to only ack when your work is complete. You can find the relevant documentation here: https://www.rabbitmq.com/confirms.html

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search