I have a requirement to read the data from redis cache in flink, but as per the requirements the cache data will be refreshed on an average every two hours. I was going through the documentation and other stackoverflow questions some people suggesting to use the apache-bahir driver to connect to redis cache but the driver is outdated and not maintained by flink. I am using richsink function in flink to connect to redis cache, but i am unable to figure out is there any way to refresh the cache from sink function. Is it recommended to use the sink function in flink to connect to redis.
2
Answers
While one of the Redis sinks may not support this out of the box, it seems like something that you may be able to explicitly handle with a process function within your pipeline that makes the appropriate API call to Redis (may vary based on your configuration).
You could implement this functionality with an explicit TTL that represents your update frequency (in this case two hours). This way as data passes through, it would check if the cache needed to be refreshed (based on a TTL) and if the state had expired, you could issue a request to refresh it that might look something like:
And this could function as a pass through prior to your sink (or if you are writing a custom sink, add this functionality to the sink):
It might prove worthwhile to build on top of the Generic Asynchronous Base Sink.