Question posted in
The official documentation can be found here.

Redis – Message Delivery Guarantee for Multiple Consumers in Pub/Sub and Messaging Queues

Requirement

A system undergoes some state change, and multiple other parts of the system has to know this(lets call them observers) so that they can perform some actions based on the current state, the actions of the observers are important, if some of the observers are not online(not listening currently due to some trouble, but will be back soon), the message should not be discarded till all the observers gets the message.

Trying to accomplish this with pub/sub model, here are my findings, (please correct if this understanding is wrong) –

The publisher creates an event on specific topic, and multiple subscribers can consume the same message. This model either provides no delivery guarantee(in redis), or delivery is guaranteed once(with messaging queues), ie. when one of the consumer acknowledges a message, the message is discarded(rabbitmq).

Example

A new Person Profile entity gets created in DB

Now,

  1. A background verification service has to know this to trigger the verification process.
  2. Subscriptions service has to know this to add default subscriptions to the user.

Now both the tasks are important, unrelated and can run in parallel.

Now In Queue model, if subscription service is down for some reason, a BG verification process acknowledges the message, the message will be removed from the queue, or if it is fire and forget like most of pub/sub, the delivery is anyhow not guaranteed for both the services.

One more point is both the tasks are unrelated and need not be triggered one after other.

In short, my need is to make sure all the consumers gets the same message and they should be able to acknowledge them individually, the message should be evicted only after all the consumers acknowledged it either of the above approaches doesn’t do this.

Anything I am missing here ? How should I approach this problem ?

Tags:

2

Answers


  1. 0

    As you mentioned it is not something that you can control with Redis Pub/Sub data structure.

    But you can do it easily with Redis Streams.

    Streams will allow you to post messages using the XADD command and then control which consumers are dealing with the message and acknowledge that message has been processed.

    You can look at these sample application that provides (in Java) example about:

    • posting and consuming messages
    • create multiple consumer groups
    • manage exceptions

    Links:

    Login or Signup to reply.
  2. 0

    This scenario is explicitly supported by RabbitMQ’s model, which separates "exchanges" from "queues":

    • A publisher always sends a message to an "exchange", which is just a stateless routing address; it doesn’t need to know what queue(s) the message should end up in
    • A consumer always reads messages from a "queue", which contains its own copy of messages, regardless of where they originated
    • Multiple consumers can subscribe to the same queue, and each message will be delivered to exactly one consumer
    • Crucially, an exchange can route the same message to multiple queues, and each will receive a copy of the message

    The key thing to understand here is that while we talk about consumers "subscribing" to a queue, the "subscription" part of a "pub-sub" setup is actually the routing from the exchange to the queue.

    So a RabbitMQ pub-sub system might look like this:

    1. A new Person Profile entity gets created in DB
    2. This event is published as a message to an "events" topic exchange with a routing key of "entity.profile.created"
    3. The exchange routes copies of the message to multiple queues:
      • A "verification_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.#"
      • A "subscription_setup_service" queue has been bound to this exchange to receive a copy of all messages matching "entity.profile.created"
    4. The consuming scripts don’t know anything about this routing, they just know that messages will appear in the queue for events that are relevant to them:
      • The verification service picks up the copy of the message on the "verification_service" queue, processes, and acknowledges it
      • The subscription setup service picks up the copy of the message on the "subscription_setup_service" queue, processes, and acknowledges it
    5. If there are multiple consuming scripts looking at the same queue, they’ll share the messages on that queue between them, but still completely independent of any other queue.

    Here’s a screenshot from this interactive visualisation tool that shows this scenario:

    RabbitMQ visualisation showing a producer, an exchange, two queues, and three consumers

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search