I have a CentOS 7 server which was running happily for 600+ days until it was rebooted recently, after which incoming web requests were receiving HTTP523 (Origin Is Unreachable) error codes (via Cloudflare, if that makes a difference?) unless I stopped the firewalld
service. Things run fine without firewalld
, but I’d rather not leave it disabled!
I’ve tried stopping docker
and firewalld
and restarting them in various sequences, but the same 523
error occurs unless I stop firewalld
.
/var/log/firewalld
contains a few warnings that might help:
WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-8acb606a3b50 -o br-8acb606a3b50 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
WARNING: AllowZoneDrifting is enabled. This is considered a n insecure configuration option. It will be removed in a future release. Please consider disabling it now.
WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target 'DOCKER':No such file or directory
WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?)
WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -F DOCKER' failed: iptables: No chain/target/match by that name.
I’ve found seemingly conflicting advice around the place regarding any manual configuration/commands required:
firewall-cmd --permanent --zone=trusted --add-interface=docker0
on a CentOS forumfirewall-cmd --zone=trusted --remove-interface=docker0 --permanent
on the offical Docker docs — surely that’s the opposite of the above?- a bunch of manual
firewall-cmd
commands on a Docker github issue — surely all of that isn’t required? - this one looks promising —
nmcli
,NetworkManager
andfirewall-cmd --permanent --zone=trusted --change-interface=docker0
I don’t fully understand where the br-8acb606a3b50
interface comes from, or whether I need to do anything to configure it as well as docker0
if I use a solution like 4.
above? It was all working fine automatically for years until the reboot!
Are some magic firewalld
incantations now required (and why?!) or is there some way I can get the system to get back into the correct auto/default configuration it was in prior to rebooting?
$ docker -v
Docker version 20.10.5, build 55c4c88
$ firewall-cmd --version
0.6.3
$ firewall-cmd --get-zones
block dmz docker drop external home internal public trusted work
2
Answers
I had some similar problems with Podman and for me i had to upgrade from Debian 9 to Debian 10 in order to fix it, because of the way firewalld handles iptables vs nftables.
To recap the chat investigation, this particular problem wasn’t related to Docker and containers. The problem was in
firewalld
not having rules forNGINX
running as a proxy for containers on the host. The solution was to add permanent firewalld rules for HTTP and HTTPS traffic:Warning messages like this one:
… can appear during normal operation, when Docker attempts to delete a rule without checking its existence first. In other words, containers can be running smoothly even when there are warnings like this.