CentOS Question posted in
The official documentation can be found here.

Kubernetes cluster with firewall enabled on CentOS(calico) not working

I’ve bringing up Kubernetes cluster with calico as CNI on CentOS 7 with firewall enabled. I’ve master and worker nodes. I was able to bring up cluster and able to list the nodes and Kubernetes system pods, all are working fine. However I’m unable to perform dns lookup.

System configuration

Kubernetes: 1.21.1
Calico: 3.19.1
Docker: 20.10.5
CentOS 7.9
IPVS enabled
Using VXLAN based network in Calico

Firewall configuration

  • The ports I’ve enabled are.

    - 6443/tcp
- 2379-2381/tcp
- 10248-10260/tcp
- 30000-32767/tcp
- 8285/udp
- 8472/udp
- 7946/udp
- 7946/tcp
- 7472/tcp
- 7472/udp
- 9100/tcp
- 443/tcp
# cAdvisor Port
- 4149/tcp
# calico
- 179/tcp
- 4789/udp
- 5473/tcp
- 9099/tcp
- 9099/udp

  • The enabled masquerade

  • Added interfaces kube-ipvs0, vxlan.calico and docker0 to trusted zone (Though was unable to add cali* interface to trusted zone as I was unable to add regex based interface rule)

The Problem is DNS lookup doesn’t work. Tried following steps to identify iptables lookup.

  • Installed DNS debugger pod
    apiVersion: v1
    kind: Pod
    metadata:
      name: dnsutils
      namespace: default
    spec:
      containers:
      - name: dnsutils
        image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
        command:
          - sleep
          - "3600"
        imagePullPolicy: IfNotPresent
      restartPolicy: Always
  • Enable firewall debugging on worker node on which pod got scheduled
    firewall-cmd --set-log-denied=all
    firewall-cmd --reload
  • Now run nslookup
    $kubectl exec -it dnsutils -- nslookup kubernetes.default

result:

   ;; connection timed out; no servers could be reached

   command terminated with exit code 1
  • Get the rule that rejected the packet
    dmesg | grep -i reject

result:

    [ 5556.708338] FINAL_REJECT: IN=calib3c61c3cba9 OUT= MAC=ee:ee:ee:ee:ee:ee:de:c8:d5:97:58:87:08:00 SRC=10.244.212.65 DST=10.96.0.10 LEN=90 TOS=0x00 PREC=0x00 TTL=64 ID=49835 PROTO=UDP SPT=52743 DPT=53 LEN=70
    [ 5561.707815] FINAL_REJECT: IN=calib3c61c3cba9 OUT= MAC=ee:ee:ee:ee:ee:ee:de:c8:d5:97:58:87:08:00 SRC=10.244.212.65 DST=10.96.0.10 LEN=90 TOS=0x00 PREC=0x00 TTL=64 ID=52640 PROTO=UDP SPT=52743 DPT=53 LEN=70
    [ 5566.708055] FINAL_REJECT: IN=calib3c61c3cba9 OUT= MAC=ee:ee:ee:ee:ee:ee:de:c8:d5:97:58:87:08:00 SRC=10.244.212.65 DST=10.96.0.10 LEN=90 TOS=0x00 PREC=0x00 TTL=64 ID=54942 PROTO=UDP SPT=52743 DPT=53 LEN=70
Tags:

2

Answers


  1. 0

    Faced a similar issue recently and it took us time to find what the actual problem is.

    Problem

    Even when you add ports for calico it does not work because, when you explicitly add port to firewall it blocks all other ports across interfaces. And calico creates a new network interface(vxlan.calico) which needs to be added to firewalld

    Symptom

    One symptom we noticed due to this was, we were able to ping pods across nodes. So icmp was working but tcp/udp did not work. Most likely you will get the same symptom.

    Solution

    You will need to add the network interface created by calico (check ifconfig to get exact name) to a trusted zone in firewall or create a new zone for it.

    Create new zone and adding interface to it:

    sudo firewall-cmd --permanent --new-zone={name}
sudo firewall-cmd --permanent --zone={name} --set-target=ACCEPT
sudo firewall-cmd --permanent --zone={name} --add-interface=vxlan.calico
sudo firewall-cmd --reload

    Note – you will need to do above steps for master as well as all nodes.

    Login or Signup to reply.
  2. 0

    I was able to get this working with the following steps.

    sudo firewall-cmd --permanent --new-zone={name}
sudo firewall-cmd --permanent --zone={name} --set-target=ACCEPT
sudo firewall-cmd --permanent --zone={name} --add-interface=vxlan.calico

    Then I looped through the calico network interfaces

    for i in $(ip a | grep cali | awk -F":" '{print $2}' | awk -F"@" '{print $1}') 
do 
  sudo firewall-cmd --permanent --zone={name} --add-interface="$i"
done
sudo firewall-cmd --reload
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search