I have a dotnet application that uses a redis cluster for caching. Recently I came across an issue where the dotnet application identifies as the cluster has gone down. but when checked with the cluster info command the cluster status is intact.
Any ideas what might have caused this issue?
TIA
–UPDATE–
2
Answers
Sometimes Redis clusters changes its configurations due to failover stuff (when you use Sentinel), it is better to instruct your code to get new config from cluster whenever its connectivity is down.
From the log, this looks like a legitimate panic at the cluster;
CLUSTERDOWNisn’t generated by the client library or your application, but rather: by the redis node that was being communicated with. This strongly suggests some major blip in the internal redis server state, which needs to be recovered before commands can be processed. This recovery could be automatic (most likely in the case of a temporary network blip) or manual (most likely in the case of critical failure potentially impacting all redundant nodes in a vertical, or at least the primary before failover election can occur).If the library failed to recover after the cluster had repaired itself, then that sounds like a potential topic for the library to investigate, but note that such recovery can never be instantaneous: when things are this ill, all the library can do is periodic polling to see if things have recovered – constantly hammering the servers with "are you happy yet? are you happy yet? are you happy yet?" would be detrimental to the cluster.