Question posted in 
Having tough time in understanding few things.
I have two replicated service host (A & B) (may be scaled to a max 3 in future) behind a gateway. Each service host needs to save some data/state (key-value) of its respective host. To avoid SPOF, the same state/data needs to be replicated to both the host so that if one server goes down then other should be in position to serve. Please suggest some mechanism to solve this problem. (to be very specific if any distributed framework)
Master-slave replication: Host A will be master and Host B will be slave.
But my question is:
- All the write request from host B will route via host A and then eventually to B.
- Also, lets say host A goes down, then there will be no write and B
will stuck till A is recovered.
Is my understanding correct that write cannot be done in slave? In case there is some workaround e.g upgrading slave to master, still re-routing issue is there as mentioned previously.
I think solution can be implemented using other approach (multi-master & leaderless) though but couldn’t come up with any conclusion. I tried comparing few of them:
- Redis – By default it is master-slave but enterprise version is
having master-master configuration also. So could be a possible
solution. - Dynamo style – Riak, Cassandra, Voldemort etc. Not sure about complexity involved
Requirement:
- Solution must be simpler and lightweight without consuming much resource of service host machine.
- Data is in key-value format
- Read/write should very fast. (in-memory would be preferable)
- Occurrence/Frequency of read/write action is not very fast.
- Volume of data is also very less. total size of data at any host at any point may not exceed 1 mb.
2
Answers
Using Redis in Cluster mode or sentinel mode will both be an option.
Redis Cluster: You can setup multiple master-slave nodes. And Redis will handle failover when master down.
Redis Sentinel: You can setup single master-slave nodes or multiple, and a sentinel cluster. When failover, sentinel will handle the new master election.
And I don’t think you need a separate database for each host of your service. Instead all hosts can connect to same cluster of Redis. When one of the host or redis node down, it will not affect your service.
Cassandra’s P2P nature gives it an obvious advantage over others where one doesn’t need to bother about how to distribute the masters and slaves in a fashion that maximum redundancy is achieved.
Using Sentinel has one disadvantage – if the Sentinel node itself fails then the entire system is affected. It is much better to scale both horizontally and vertically i.e. multi-master multi-slave setup with the masters and slaves distributed across nodes.