Question 121764 in Redis Question posted in
The official documentation can be found here.

Handle redis connection reset

We are facing [Errno 54] Connection reset by peer at very random in our application and looks like it is being triggered by redis server than client. Python’s redis client have backoff strategy implementation but it’s unable to handle this scenario.

There are github issues on official repo as well and many people commented recently confirming this problem.

Step to reproduce

$ ipython
# in python client
import redis
from redis.retry import Retry
from redis.exceptions import (TimeoutError, ConnectionError)
from redis.backoff import ExponentialBackoff

# connect client with exponential backoff retry
client = redis.StrictRedis(retry=Retry(ExponentialBackoff(cap=10, base=1), 25), retry_on_error=[ConnectionError, TimeoutError, ConnectionResetError], health_check_interval=1)

client.keys()
# print all keys

Now reset connection directly from redis server

$ redis-cli
RESET

Wait for 120 second or more and run client again

# in client
client.keys()
---------------------------------------------------------------------------
ConnectionResetError                      Traceback (most recent call last)
<ipython-input-91-011ce9f936fc> in <module>
----> 1 client.keys("rq*")

~/path-to-python-env/env/lib/python3.8/site-packages/redis/commands/core.py in keys(self, pattern, **kwargs)
   1386         For more information check https://redis.io/commands/keys
   1387         """
-> 1388         return self.execute_command("KEYS", pattern, **kwargs)
   1389 
   1390     def lmove(self, first_list, second_list, src="LEFT", dest="RIGHT"):

~/path-to-python-env/env/lib/python3.8/site-packages/redis/client.py in execute_command(self, *args, **options)
   1168         pool = self.connection_pool
   1169         command_name = args[0]
-> 1170         conn = self.connection or pool.get_connection(command_name, **options)
   1171 
   1172         try:

~/path-to-python-env/env/lib/python3.8/site-packages/redis/connection.py in get_connection(self, command_name, *keys, **options)
   1315             # closed. either way, reconnect and verify everything is good.
   1316             try:
-> 1317                 if connection.can_read():
   1318                     raise ConnectionError("Connection has data")
   1319             except ConnectionError:

~/path-to-python-env/env/lib/python3.8/site-packages/redis/connection.py in can_read(self, timeout)
    793         if not sock:
    794             self.connect()
--> 795         return self._parser.can_read(timeout)
    796 
    797     def read_response(self, disable_decoding=False):

~/path-to-python-env/env/lib/python3.8/site-packages/redis/connection.py in can_read(self, timeout)
    315 
    316     def can_read(self, timeout):
--> 317         return self._buffer and self._buffer.can_read(timeout)
    318 
    319     def read_response(self, disable_decoding=False):

~/path-to-python-env/env/lib/python3.8/site-packages/redis/connection.py in can_read(self, timeout)
    222 
    223     def can_read(self, timeout):
--> 224         return bool(self.length) or self._read_from_socket(
    225             timeout=timeout, raise_on_timeout=False
    226         )

~/path-to-python-env/env/lib/python3.8/site-packages/redis/connection.py in _read_from_socket(self, length, timeout, raise_on_timeout)
    192                 sock.settimeout(timeout)
    193             while True:
--> 194                 data = self._sock.recv(socket_read_size)
    195                 # an empty string indicates the server shutdown the socket
    196                 if isinstance(data, bytes) and len(data) == 0:

ConnectionResetError: [Errno 54] Connection reset by peer

Running it second time works perfectly, though it should have taken care by Retry strategy.

client.keys()
# prints all keys

configuration

redis server - 6.2.6
python redis - 4.1.0

We can write our own try/catch around redis client, but we are using some libraries like rq and flask-cache which uses redis internally and have no interface to modify it’s flow.

Any help is much appreciated.

Tags:

3

Answers


  1. Chosen as BEST ANSWER
    0

    I end up writing a custom class of Redis which retries on failure. Sharing it here in case anyone finds it useful.

    import redis
import logging

from retry import retry

logger = logging.getLogger(__name__)


class RedisCustom(redis.Redis):
    def __init__(self, *args, **kwargs):
        super(RedisCustom, self).__init__(*args, **kwargs)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def set(self, key, value, ex=None, px=None, nx=False, xx=False):
        return super(RedisCustom, self).set(key, value, ex, px, nx, xx)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def get(self, name):
        return super(RedisCustom, self).get(name)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def delete(self, *names):
        return super(RedisCustom, self).delete(*names)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def hset(self, name, key=None, value=None, mapping=None):
        return super(RedisCustom, self).hset(name, key, value, mapping)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def hmset(self, name, mapping):
        return super(RedisCustom, self).hmset(name, mapping)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def hgetall(self, name):
        return super(RedisCustom, self).hgetall(name)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def scan_iter(self, match=None, count=None, _type=None, **kwargs):
        return super(RedisCustom, self).scan_iter(match, count, _type, **kwargs)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def hdel(self, name, *keys):
        return super(RedisCustom, self).hdel(name, *keys)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def keys(self, pattern="*", **kwargs):
        return super(RedisCustom, self).keys(pattern, **kwargs)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def info(self, section=None, **kwargs):
        return super(RedisCustom, self).info(section, **kwargs)

    @retry(ConnectionResetError, delay=0.1, tries=5)
    def ping(self, **kwargs):
        return super(RedisCustom, self).ping(**kwargs)

  2. 0

    Try using a connection Pool as it will keep alive and automatically attempt to recover in the background. In the REPL

    import redis
pool = redis.ConnectionPool(host="localhost", port=6379)
r = redis.Redis(connection_pool=pool)
r.ping()

    In another shell you can test by running redis-cli

    client LIST
client kill IP:PORT

    Then run the

    r.ping

    And you should see a new connection in the redis-cli

    client LIST
    Login or Signup to reply.
  3. 0

    You can also set a healtcheck value,

    client = redis.Redis(..., health_check_interval=30)

    it will reestablish the connection.
    More details in https://github.com/redis/redis-py/issues/1186

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search