Question posted in
The official documentation can be found here.

Redis – Distributed Atomic Counter

I have a service built on Microservices architecture. The design includes 3 components that communicating with queues. System components:

  1. Microservice “A” gets a request and generates subrequests. Each sub-request is a queue message written to the Service “B” queue. Each incoming message producing multiple messages to queue “B”.
  2. Microservice “B” gets a task to execute and execute it. When all the subrequests, made by service “A”, executed – this service enqueues a message into service “C” queue. Enqueuing a message into the next queue happens only when all the execution of the messages is done.
  3. Microservice “C” get a message and mark the row in the database as completed.

They way it’s implemented today is the on step “1” I create a counter with the init value of the number of tasks. Each execution of service “B” is decrement this counter by 1. After the last execution – the counter value will be 0 – then the execution thread will generate a new message to service “C”. This process ensure only 1 message into queue “C”.

Basicly, it’s working. The problem with this, is the number of decrement operations.

Today I implemented this counter as a database row (SQL update operation is atomic). It’s reliable but make my database hard life.

Other solutions I thought about:

  1. MySQL Database: reliable, relatively slow. Limited throughput.
  2. Redis: work fast, but we cannot trust it (after restart all the data is removed).
  3. NoSQL: not handling concurrency in a good way. Hotspots break the performances.

Wondering if this kind of problem has a better solution?

Tags:

2

Answers


  1. 0

    Check out “Atomic Long” from Apache Curator/Zookeeper or Apache ignite.

    Login or Signup to reply.
  2. 0

    In Java I would use a Countdown latch.

    But since your problem is distributed you need to take another approach.

    You need to atomically update a number, if you take the Zookeeper approach you can implement it like this

    Imagine you have 9 servers and you try decrement the number, you need to wait until 4 nodes approve the change before it is accepted. The database solution you picked might be fine as the database server enforces a total order over the decrements and most database servers support serialisable and linearisable transaction processing. So if two of your processes try to decrement in parallel one shall be aborted due to concurrency control (locks or timestamps)

    With zookeeper there is one leader so all requests or messages are serialised and given a transaction Identifier zxid this is monotonically increasing number. If two decrements get received then they shall be executed in serial even if they were done in parallel. This is kind of what you want.

    Another alternative is to use a CRDT. You have a grow only set and at the beginning you store the process name and a flag.

    Whenever an app finished you merge a record into the CRDT with the process name and the flag set to true.

    When the number of true flags is equal to the number of processes you can do the next step.

    Here is some documentation on curator project that talks of an atomic long implementation.
    https://github.com/Netflix/curator/wiki/Distributed-Atomic-Long/d0cd56ce402376263085b95a06d5dcb841ed792e

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search