Question posted in
The official documentation can be found here.

Redis – Scan keys with lua script in Node

I have been trying to get all the keys from Redis using scan while also avoiding using count. Problem is, when there are many keys (ex. 1000000) scans SCAN 0 MATCH * are much slower than:

KEYS *

or

SCAN 0 MATCH * COUNT 1000000

I have read an article saying to use Lua for better performance. Can anybody help me regarding this matter? I want to use Lua to handle a scan of all the keys where the number of keys is more than 1M. I am using the redis package by npm.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Found related solution. Please go through this link. Using of Keys is not advised and also if you go through the documentation you can see that. Though here is the performance is given but it's for the sake of understanding more deep knowledge, nothing else. The best way is to use Lua Script which is advisable and also the performance is really good. And if you see the comments you will also get to know about the difference as well, I guess. The performance for the KEY, SCAN , Lua Script :

    • KEYS: the fastest method
    • SCAN: ~1.5 times slower
    • SCAN with lua:~2.4 times slower
    • SCAN with lua and additional checks: ~4.3 times slower
    • KEYS with lua: ~18 times slower

    For me, it was quite difficult to understand the syntax of Lua. So, here is the code to help understanding the syntax.

    Scan without count in Lua Script: 

    local ans, has, cursor = {}, {}, "0";
repeat
    local t = redis.call("SCAN", cursor, "MATCH", KEYS[1], "COUNT", 1000000000);
    local list = t[2];
    for i = 1, #list do
        local s = list[i];
        if has[s] == nil then has[s] = 1; ans[#ans + 1] = s; end;
    end;
    cursor = t[1];
until cursor == "0";
return #ans; --or return ans;

  2. 0

    Using SCAN with COUNT 1000000 would be slightly worst than using KEYS.

    Redis is single-threaded. One of the reasons SCAN was introduced is to allow going through all the keys without blocking the server for a long time, by going a few steps at a time.

    The default value is 10. It means the command will bring back more or less 10 keys, could be less if the keys are sparsely populated in the hash slots, or filtered out by the MATCH pattern. It could be more if some keys are sharing a hash slot. Anyhow, the work performed is proportional to the COUNT parameter.

    This doesn’t mean the KEYS command (or a SCAN with a big COUNT number) is forbidden. If you are ok with your server being blocked for the time it takes to run, go ahead.

    See Is there any recommended value of COUNT for SCAN / HSCAN command in REDIS? for more details and a Lua example.

    In It is possible to merge two separate DBs (e.g. db0, db1) running in one single Redis instance (6379) into one single DB (db0)? and How can I get all of the sets in redis? I have posted examples of how to use SCAN within a Lua script to achieve custom requirements.

    You may find Redic `SCAN`: how to maintain a balance between newcomming keys that might match and ensure eventual result in a reasonable time? interesting.

    How to use Lua in Redis with node.js: executing redis eval command to run Lua script in nodeJS

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search