I would like my application to do something whenever it receives an invalidation message from Redis. But as is the case for any distributed system, there’s intrinsic risk in network packet drops etc.
It’d be great if my application (i.e. Redis client-side) detects when an invalidation message fails to reach it. But I think this is impossible by definition – messages are sent by Redis server and if they fail to reach the client, then clients won’t hear anything about it. Or, at the very least, does Redis server-side expose any metrics to indicate how often it fails to send invalidation messages?
The closest thing I’ve seen Redis discuss is (source: Client-side caching reference – What to do when losing connection with the server):
Make sure that if the connection is lost, the local cache is flushed
And the client library I’m using, rueidis, does implement this. Can I assume that as long as the connection is live, 100% invalidation messages will always be received?
References (generally useful, but none of them explicitly discussed my question):
2
Answers
The official Redis client libraries Jedis and redis-py, which support client-side caching, will flush the local cache in case a single connection or a connection in the pool is disconnected.
Documentation now reports the behavior.
Rueidis clears the local cache on disconnections.
Since the invalidations are delivered on the same client tcp connection, they are guaranteed to be delivered but may be delayed for various reasons. Rueidis uses periodic background pings to mitigate the delay on a tcp connection. It also requires you to specify a local TTL on the cache entry to further reduce the chance of getting staled data.
To monitor the delay, I think you can watch the client_longest_output_list metric to see if it keeps growing.