Question 121362 in Redis Question posted in
The official documentation can be found here.

Redis – Running tally lua script periodically via remote script

I am setting up a Redis instance that receives datapoints consisting of a key/value pair, the key can have various prefixes to separate types. Each key have a ttl expiration. Every 15 min I plan to have a lua script running that will do a tally of the various values of keys with a specific prefix. This is being kicked off from a cron job running a node script.

local data = redis.call('KEYS', KEYS[1] .. "*")
for i=1,#data do
  local value = 'rollup.' .. KEYS[1] .. redis.call('GET', data[i])
  if redis.call('EXISTS', value)==1 then
    redis.call('INCR', value)
  else
    redis.call('SET', value, 1)
  end
end

node script:

cron.scheduleJob("*/15 * * * *", async () => {
   try {
     await redisServer.eval(fs.readFileSync("./dist/tally.lua"), 0);
   } catch (ex) {
     logger.error("Calling rollup lua failed. " + util.inspect(ex));
   }
 });

I seem to be getting these errors

ReplyError: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.

periodically. Is this a problem with the lua script or is there a better way that I can run this? i.e. other than running an external lua script being run on the same port that receives other regular calls like set, expires and ping (I also have a heartbeat that pings redis every second to make sure it is a live). I have read Redis has a way to schedule tasks, however I am having a hard time finding examples of how this is done. Or is there maybe a build-in way of achieving what I am trying to do?

EDIT: My fundamental problem is likely that the lua script is running atomically, i.e. nothing else can run while this is happening, causing everything else interacting with Redis to back up. Are there a faster way to do a tally, potentially one that does not involve a lua script, should I pull the data and do the tally in the node script?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Instead of using a script to actively tally the data, I decided to increment the tally when inserting the key/value pair with a ttl, subscribe to the expiration of the key and then decrement it when it expires at that point. Adding this answer in case anybody is trying to do something similar.


  2. 0

    The use of Keys is an anti-pattern

    It’s considered a real anti-pattern to use the Keys command anywhere in Redis, Keys will literally block redis while it traverses the entire key-space which can take quite a while.

    And it’s killing your script

    What’s happening here is that your lua script is exceeding the lua-time-limit (defaults to 5 seconds) which is the maximum allowed time before redis starts rejecting commands because of a blocking lua script.

    Basically, you’re running a bad script and it’s blocking Redis from doing anything.

    Use a sorted set instead

    This seems like something that would be better handled with a sorted set. You can of course keep the TTLs on the keys that you want to expire, that much is fine, but realistically, you could set in a sorted set, and have the expiration times be the score and the member be the key-name

    so if you do a:

    SET foo bar EX 50

    You would just compute the UNIX timestamp of 50 seconds from now, and use that as the score, let’s just say that’s 1659120294, you’d then add to your rollup-zset:

    ZADD zset:rollup 1659120294 foo

    Then, all you need to do when you query the count is use a count from the current time, let’s say that occurs at 1659120394:

    ZCOUNT zset:rollup 1659120394 inf

    That will render the count of all the un-expired keys

    the best part is that when you are finished counting, you can just go in in one shot and delete all the expired keys out of your sorted set, with a ZREMRANGEBYSCORE, assuming the same time

    ZREMRANGEBYSCORE zsetrollup -inf 1659120394

    ZADD and ZCOUNT are very fast, they are O(1) and O(log(n)) respectively, ZREMRANGE is a bit slower where it’s O(N) in the number of elements you are purging from it every time. However, this is going to be relatively slight compared to running keys against your database every 15 min.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search