Question posted in
The official documentation can be found here.

Redis – HMGET and correlation beetwen number of fields and reading performance

Have some question about HMGET performance.

Can you explain how number of fields in hash affect reading performance?

Example 1. 20000 requests. HMGET with 4 fields. Commands in pipeline. Hash contains 760 fields
On test machine it takes about 1500ms.

Example 2. 20000 requests. HMGET with 4 fields. Commands in pipeline. Hash contains 30 fields
On test machine it takes about 300ms.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I did similar test to @LeoMurillo did.

    redis-benchmark -n 200000 -c 1 -q HMGET Test1HMGET 1 2 3 4
HMGET Test1HMGET 1 2 3 4: 26648.90 requests per second

redis-benchmark -n 200000 -c 1 -q HMGET Test1HMGET 10 9 8 7
HMGET Test1HMGET 10 9 8 7: 26267.40 requests per second

redis-benchmark -n 200000 -c 1 -q HMGET Test2HMGET 1 2 3 4
HMGET Test2HMGET 1 2 3 4: 26737.97 requests per second

redis-benchmark -n 200000 -c 1 -q HMGET Test2HMGET 1000 999 998 997
HMGET Test2HMGET 1000 999 998 997: 11201.97 requests per second

redis-benchmark -n 200000 -c 1 -q HMGET Test3HMGET 1 2 3 4
HMGET Test3HMGET 1 2 3 4: 27514.10 requests per second

redis-benchmark -n 200000 -c 1 -q HMGET Test3HMGET 100000 99999 99998 99997
HMGET Test3HMGET 100000 99999 99998 99997: 27311.21 requests per second

    Then I checked hash-max-ziplist-entries. It is set to 1500. So in my case Test1HMGET and Test2HMGET have encoding:ziplist, and Test3HMGET encoding:hashtable.

    So I think it is all about compression. When a hashset is compressed, getting fields from the end of a set is so much slower.


  2. 0

    It should be O(1) in both cases – independent of the number of fields in the hash. So 5x the time for 20x the number of keys sounds excessive.

    I tried to replicate and got very consistent performance. Here what I did:

    Created three hashes, with 10, 1000, 100000 fields each:

    > EVAL "for i=1,10 do redis.call('HSET', KEYS[1], i, i) end" 1 Test1HMGET
(nil)
> EVAL "for i=1,1000 do redis.call('HSET', KEYS[1], i, i) end" 1 Test2HMGET
(nil)
> EVAL "for i=1,100000 do redis.call('HSET', KEYS[1], i, i) end" 1 Test3HMGET
(nil)
> HLEN Test1HMGET
(integer) 10
> HLEN Test2HMGET
(integer) 1000
> HLEN Test3HMGET
(integer) 100000

    Then run redis-benchmark to test HMGET:

    $ redis-benchmark -n 200000 -c 1 -q HMGET Test1HMGET 4 6 7 9
HMGET Test1HMGET 4 6 7 9: 8841.73 requests per second
$ redis-benchmark -n 200000 -c 1 -q HMGET Test2HMGET 4 6 7 9
HMGET Test2HMGET 4 6 7 9: 8788.89 requests per second
$ redis-benchmark -n 200000 -c 1 -q HMGET Test3HMGET 4 6 7 9
HMGET Test3HMGET 4 6 7 9: 8863.68 requests per second
$ redis-benchmark -n 200000 -c 1 -q HMGET Test3HMGET 45 667 567 56789
HMGET Test3HMGET 45 667 567 56789: 8819.51 requests per second

    As shown, regardless of the hash length, it consistently got ~8800 rps.

    Other commands, like HGETALL, do show a decline.

    $ redis-benchmark -n 20000 -c 1 -q HGETALL Test1HMGET
HGETALL Test1HMGET: 7840.06 requests per second
$ redis-benchmark -n 20000 -c 1 -q HGETALL Test2HMGET
HGETALL Test2HMGET: 600.08 requests per second

    There must be something else on your test driving the difference you observe, perhaps payload difference if you are using different fields.

    Update: As pointed out by the OP, there is a performance difference once you get close to hash-max-ziplist-entries. See Antirez article, and this post. Your average read performance decay, due to compression. It is a CPU / memory trade-off.

    On my machine, I had max-ziplist-entries = 512. Performance is 2.3x slower towards the end of the key-space. Once we go over the compression optimization threshold (514 fields), it is constant across the key-space, but memory usage went up 6.6x.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search