Question posted in
The official documentation can be found here.

Python 2.7 and 3.8.0 compatible django-redis serializer

I Have two server running one is running on python2.7 and another on python3.8.

for both the servers we have the common Django-cache server.

python2.7 server is setting up cache in the Django-cache and python3.8 trying to read this throwing an error saying

File “/usr/local/bin/python-3.8.0/lib/python3.8/site-packages/django_redis/serializers/pickle.py”, line > 35, in loads
return pickle.loads(value)
UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xe5 in position 1: ordinal not in range(128)

i already read the below post with the same question link

my goal here is to able to read and from both versions of python to the common django-cache.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    The solution which worked for me is we implemented a custom serializer and adder that in the redis_cache config.

    import six
from django.utils.encoding import force_bytes
from django_redis.serializers.pickle import PickleSerializer

try:
    import cPickle as pickle
except ImportError:
    import pickle


class CcustomPickleSerializer(PickleSerializer):

    def loads(self, value):
        if six.PY3:
            return self._loads_py3(value)
        return super().loads(force_bytes(value))

    def _loads_py3(self, value):
        return pickle.loads(
            force_bytes(value),
            fix_imports=True,
            encoding='latin1'
        )

    if you use the encoding method as 'bytes'. you can get bytes else you will get a string.

    and the below-mentioned line in the cache config inside the settings.py.

    'SERIALIZER':'file.location.CustomPickleSerializer' in.

    
CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379/0',
        'OPTIONS': {
            'KEY_PREFIX': 'personify',
            'SERIALIZER':'file.location.CustomPickleSerializer',
            'PARSER_CLASS': 'redis.connection.HiredisParser',  # Hiredis is WAY faster than redis-py
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            'CONNECTION_POOL_KWARGS': {
                'max_connections': 100
            },
            'PICKLE_VERSION': 2,  # Make sure we're using v2 of pickle, which is pretty efficient.
            'SOCKET_CONNECT_TIMEOUT': 5,  # Seconds (timeout for the connection to be established).
            'SOCKET_TIMEOUT': 5,  # Seconds (timeout for read and write operations after the connection is established).
            'IGNORE_EXCEPTIONS': False  # Set to True to act like memcached - i.e. don't raise errors.
        }
    }
}

  2. 0

    Default str encoding in python 2.7 is ASCII while the default for 3.x is utf-8 so you need to consider this.

    If you look closely it’s not Redis issue, it’s encoding issue, hence the error message:

    UnicodeDecodeError: ‘ascii’ codec can’t decode byte

    To solve this problem you have to set the default encoding to utf-8 in your python 2.7 program. 

    import sys  

reload(sys)  
sys.setdefaultencoding('utf8')

    There are more details about the python encoding conversion issue here: How to fix UnicodeDecoderError

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search