Question posted in
The official documentation can be found here.

Redis – Understanding why you would want to process Message Queues at a future time

So I’m trying to understand what practical problems Queues solve. By reading all the information from Google, I get the high-level.

  • Push message to Queue for processing at a later time

So I’m looking at an architecture from Company A and they have different use cases for Job Queueing like for example

  • chat messages
  • file conversion
  • searching
  • Heavy sql queries

Why process it at a later time?

Here’s my best guess…

  1. Let’s say I have an application that can process 10 “things” at a time.
  2. My application then maxes out it’s processing capacity.
  3. an 11th request came in so app puts it in the Queue for later processing

Assuming this is a valid Use Case, wouldn’t adding more servers to process more “things” make sense? Is it because it’s more costly to add more servers than employ a Queue and sacrifice response time a little bit?

Given my Use Case examples, what other problems would Queues solve for them?

Tags:

2

Answers


  1. 0

    Have you ever lined up at a bank when it is busy? You would have waited in a queue.

    “But,” you could say, “wouldn’t adding more staff to process more customers make sense? Is it because it’s more costly to add more staff than employ a Queue and sacrifice response time a little bit?”

    That would be correct. It can be quite costly to staff a bank based on the peak number of customers who would arrive each day. It is cheaper to staff below this level and have some customers wait in a queue.

    Also, the number of customers each day are not 100% predictable. A queue allows excess demand to wait without breaking the system.

    Queues enable decoupling.

    For example, imagine an online store where customers purchase an item. They select the item, provide a credit card number and click ‘Purchase’. If the credit card is declined, the online store can immediately prompt them to re-enter the number. This interaction has to take place immediately while the customer is still online.

    However, there is no need to have the customer wait while an invoice is generated, a record is added to the accounting system and inventory is pulled off the shelf. This can be decoupled from the ordering process. A good way to do it is to push the order into a queue, which can be handled by the next system.

    If that ‘next system’ happens to be offline at the moment, there is no reason to cancel the whole sale. The transaction can be processed when the ‘next system’ comes back online. This is much better than failing the whole process just because one component (which is not required immediately) has a failure.

    Bottom line: Queues are excellent. They enable better handling of failures. They makes things more resilient (just wait a few minutes and try again!). They should be used at all times when the process is compatible with a queuing architecture.

    Login or Signup to reply.
  2. 0

    Let’s do scenarios

    Scenario 1 without queue:

    • you request an endpoint /blabla/do-eveything/

    • this request do

      1. download an image from very slow FTP
        e.g 1.5 sec (can error, retry ? add +X sec)

      2. attach the image to an email

      3. send an email (3 sec)
        e.g 1 sec (can error, retry ? add +X sec)

      4. confirmation received > store confirmation to a third company tracking stuff
        e.g 1.5 (can error, retry ? add +X sec)

      5. when tracking confirm, update your data from another third company for big data purpose
        e.g 2 sec (can error, retry ? add +X sec)

      6. … you get the idead

      7. return the response e.g 11 sec later (this is to slow) or more or timeout when everything failed

    End user said internet was faster 20 years ago, maybe I need to change my internet connection or change my 16 threads

    Scenario 2 queue everything you can:

    • you request an endpoint /blabla/do-eveything/

    • this request do

      1. Queue job “DO_EVERYTHING”
        e.g 0.02 sec

      2. Return the response less then 0.250 sec

    End user said that is website/app is too fast, I can keep my 56K internet connection

    on queue/event system one failed job can be retry later without affeting the end user
    you can pause job, add a unlimited number a task/step after the original message
    better fault tolerance

    Working with queue will allow you a better micro/nano service architecture, better testing because, you can test a single job, intead of a full controller that do everything…

    Ye, is maybe more work, more thinking, but a the end no need to think about the work when holidays

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search