Question posted in
The official documentation can be found here.

Redis – How do I execute kubelet –resolve-conf?:

I am running into an issue where coredns doesn’t seem to be working on my ubuntu instance, I get allot of the following errors from the logs:

2020-07-16T05:28:36.648Z [INFO] 172.17.0.8:60594 - 29368 "AAAA IN redis-master.cluster.local. udp 44 false 512" NXDOMAIN qr,rd 137 0.000053955s
2020-07-16T05:28:36.650Z [INFO] 172.17.0.8:43585 - 56935 "AAAA IN redis-master. udp 30 false 512" NXDOMAIN qr,rd,ra 30 0.001305981s
2020-07-16T05:28:36.650Z [INFO] 172.17.0.8:43585 - 55951 "A IN redis-master. udp 30 false 512" NXDOMAIN qr,rd,ra 30 0.001790145s
2020-07-16T05:28:36.803Z [INFO] 172.17.0.11:60313 - 14198 "A IN redis-master.cluster.local. udp 44 false 512" NXDOMAIN qr,rd 137 0.000083909s
2020-07-16T05:28:36.803Z [INFO] 172.17.0.11:60313 - 14670 "AAAA IN redis-master.cluster.local. udp 44 false 512" NXDOMAIN qr,rd 137 0.000078636s
2020-07-16T05:28:36.808Z [INFO] 172.17.0.11:42385 - 5737 "A IN redis-master.svc.cluster.local. udp 48 false 512" NXDOMAIN qr,rd 141 0.000095061s
2020-07-16T05:28:36.808Z [INFO] 172.17.0.11:42385 - 6139 "AAAA IN redis-master.svc.cluster.local. udp 48 false 512" NXDOMAIN qr,rd 141 0.000088591s
2020-07-16T05:28:36.809Z [INFO] 172.17.0.11:47547 - 18338 "A IN redis-master.cluster.local. udp 44 false 512" NXDOMAIN qr,rd 137 0.000077331s

I’m hoping that the following paragraph from the docs found here is the solution:

Some Linux distributions (e.g. Ubuntu) use a local DNS resolver by
default (systemd-resolved). Systemd-resolved moves and replaces
/etc/resolv.conf with a stub file that can cause a fatal forwarding
loop when resolving names in upstream servers. This can be fixed
manually by using kubelet’s –resolv-conf flag to point to the correct
resolv.conf (With systemd-resolved, this is
/run/systemd/resolve/resolv.conf). kubeadm automatically detects
systemd-resolved, and adjusts the kubelet flags accordingly.

However, I lack an understanding of how to execute the call to the kubelet. I could use an explanation or code example of how it is supposed to be done.

I am not sure if it is relevant but I am using minikube to test my cluster.

Thanks in advance for any help,
Cheers!

Tags:

4

Answers


  1. 0

    Sounds like a conflict with systemd-resolved and coredns in your minikube. You can try disabling systemd-resolved since you didn’t create your cluster using kubeadm:

    • Disable systemd-resolved ⌨💣

      sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved

    • Put this line in the [main] section of your /etc/NetworkManager/NetworkManager.conf ⌨:

      dns=default

    • Delete the symlink /etc/resolv.conf ⌨💣

      sudo rm /etc/resolv.conf

    • Restart NetworkManager ⌨🏃‍♀️🏃‍♀️🏃‍♀️🏃🏃🏃

      sudo systemctl restart NetworkManager
    Login or Signup to reply.
  2. 0

    I experienced this issue today, this is what I found – am still validating this answer.

    https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

    Create the following file for every host running kubelet.

    /etc/systemd/system/docker.service.d/10-docker-opts.conf

    [Service]
MountFlags=shared
Environment="DOCKER_OPTS=--resolv-conf=/run/systemd/resolve/resolv.conf"
    Login or Signup to reply.
  3. 0

    Although original question was about minikube cluster, here is a couple of methods for cluster created by using kubeadm.

    First method, using deprecated as of time writing flag –resolv-conf

    Source of information https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/#the-kubelet-drop-in-file-for-systemd

    • sudo create file /etc/default/kubelet with following content

      KUBELET_EXTRA_ARGS=--resolv-conf=path_to_your_resolv.conf

    • restart kubelet service sudo systemctl restart kubelet

    • recreate CoreDNS pods (restart rollout or delete existing pods)

    Second uses KubeletConfiguration object

    Source https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure/#updating-the-kubeletconfiguration

    • edit kubelet config stored in cluster kubectl edit cm -n kube-system kubelet-config, look for resolvConf: ***
    • then download it from cluster to local config on node kubeadm upgrade node phase kubelet-config
    • restart kubelet service sudo systemctl restart kubelet
    • recreate CoreDNS pods (restart rollout or delete existing pods)
    Login or Signup to reply.
  4. 0

    Actually, like Alexander wrote (the second approach) works, but in my case I didn’t have to edit anything in kublet config (it was already pointing to the right /run/systemd/resolve/resolv.conf) but only do last three steps, so:

    • kubeadm upgrade node phase kubelet-config
    • sudo systemctl restart kubelet
    • kubectl -n kube-system delete pod/coredns-<id> ...

    Please bear in mind that this issue is frequent on ubuntu-based nodes, source

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search