Different DNS resolution inside cluster - Debian

TomHapl
January 20, 2022
175 views
1 vote
2 Answers

I have 2 containers – busybox and debian:stretch. Both have the same content in /etc/resolv.conf and etc/hosts and ClusterFirst option.

I am using Rancher version 2.5.7 with kubernetes 1.20.5. Both containers are on the same host:

Busybox:

$ ping example.com
PING example.com (93.184.216.34): 56 data bytes
64 bytes from 93.184.216.34: seq=0 ttl=49 time=107.441 ms
64 bytes from 93.184.216.34: seq=1 ttl=49 time=109.022 ms
64 bytes from 93.184.216.34: seq=2 ttl=49 time=113.877 ms
64 bytes from 93.184.216.34: seq=3 ttl=49 time=107.547 ms
64 bytes from 93.184.216.34: seq=4 ttl=49 time=112.040 ms
64 bytes from 93.184.216.34: seq=5 ttl=49 time=110.508 ms
64 bytes from 93.184.216.34: seq=6 ttl=49 time=107.892 ms

Debian:

root@debian-7bg8bfd98c-ft6t9:$ ping example.com
PING example.com (85.93.165.117): 56(84) data bytes
64 bytes from web18.profiwh.com seq=1 ttl=62 time=0.559 ms
64 bytes from web18.profiwh.com seq=2 ttl=62 time=0.507 ms
64 bytes from web18.profiwh.com seq=3 ttl=62 time=0.658 ms
64 bytes from web18.profiwh.com seq=4 ttl=62 time=0.914 ms

--- profiwh.com ping statistics ---
4 packets transwmitted, 4 received, 0% packet loss, time 3013ms
rtt min/avg/max/mdev = 0.507/0.657/0.914/0.159 ms

Busybox resolves example.com to: 93.184.216.34 (the correct one).

Debian resolves example.com to: 85.93.165.117 which is probably my provider.

Busybox deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    field.cattle.io/creatorId: user-5srkl
  creationTimestamp: "2022-01-20T21:59:53Z"
  generation: 1
  labels:
    cattle.io/creator: norman
    workload.user.cattle.io/workloadselector: deployment-obchod-uat-busybox
  managedFields:
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:field.cattle.io/creatorId: {}
        f:labels:
          .: {}
          f:cattle.io/creator: {}
          f:workload.user.cattle.io/workloadselector: {}
      f:spec:
        f:progressDeadlineSeconds: {}
        f:replicas: {}
        f:revisionHistoryLimit: {}
        f:selector: {}
        f:strategy:
          f:rollingUpdate:
            .: {}
            f:maxSurge: {}
            f:maxUnavailable: {}
          f:type: {}
        f:template:
          f:metadata:
            f:annotations:
              .: {}
              f:cattle.io/timestamp: {}
            f:labels:
              .: {}
              f:workload.user.cattle.io/workloadselector: {}
          f:spec:
            f:containers:
              k:{"name":"busybox"}:
                .: {}
                f:image: {}
                f:imagePullPolicy: {}
                f:name: {}
                f:resources: {}
                f:securityContext:
                  .: {}
                  f:allowPrivilegeEscalation: {}
                  f:capabilities: {}
                  f:privileged: {}
                  f:readOnlyRootFilesystem: {}
                  f:runAsNonRoot: {}
                f:stdin: {}
                f:terminationMessagePath: {}
                f:terminationMessagePolicy: {}
                f:tty: {}
            f:dnsPolicy: {}
            f:imagePullSecrets:
              .: {}
              k:{"name":"dockerhub"}:
                .: {}
                f:name: {}
            f:restartPolicy: {}
            f:schedulerName: {}
            f:securityContext: {}
            f:terminationGracePeriodSeconds: {}
    manager: rancher
    operation: Update
    time: "2022-01-20T21:59:53Z"
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:deployment.kubernetes.io/revision: {}
      f:status:
        f:availableReplicas: {}
        f:conditions:
          .: {}
          k:{"type":"Available"}:
            .: {}
            f:lastTransitionTime: {}
            f:lastUpdateTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"Progressing"}:
            .: {}
            f:lastTransitionTime: {}
            f:lastUpdateTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
        f:observedGeneration: {}
        f:readyReplicas: {}
        f:replicas: {}
        f:updatedReplicas: {}
    manager: kube-controller-manager
    operation: Update
    time: "2022-01-20T22:00:24Z"
  name: busybox
  namespace: obchod-uat
  resourceVersion: "56780865"
  uid: 3463ee83-a102-4842-a1bc-6939683d7807
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      workload.user.cattle.io/workloadselector: deployment-obchod-uat-busybox
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        cattle.io/timestamp: "2022-01-20T21:59:52Z"
      creationTimestamp: null
      labels:
        workload.user.cattle.io/workloadselector: deployment-obchod-uat-busybox
    spec:
      containers:
      - image: busybox
        imagePullPolicy: Always
        name: busybox
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities: {}
          privileged: false
          readOnlyRootFilesystem: false
          runAsNonRoot: false
        stdin: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        tty: true
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: dockerhub
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-01-20T22:00:24Z"
    lastUpdateTime: "2022-01-20T22:00:24Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-01-20T22:00:18Z"
    lastUpdateTime: "2022-01-20T22:00:24Z"
    message: ReplicaSet "busybox-55cf5cdbb8" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Debian deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    field.cattle.io/creatorId: user-5srkl
  creationTimestamp: "2022-01-20T22:35:09Z"
  generation: 1
  labels:
    cattle.io/creator: norman
    workload.user.cattle.io/workloadselector: deployment-obchod-uat-debian
  managedFields:
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:field.cattle.io/creatorId: {}
        f:labels:
          .: {}
          f:cattle.io/creator: {}
          f:workload.user.cattle.io/workloadselector: {}
      f:spec:
        f:progressDeadlineSeconds: {}
        f:replicas: {}
        f:revisionHistoryLimit: {}
        f:selector: {}
        f:strategy:
          f:rollingUpdate:
            .: {}
            f:maxSurge: {}
            f:maxUnavailable: {}
          f:type: {}
        f:template:
          f:metadata:
            f:annotations:
              .: {}
              f:cattle.io/timestamp: {}
            f:labels:
              .: {}
              f:workload.user.cattle.io/workloadselector: {}
          f:spec:
            f:containers:
              k:{"name":"debian"}:
                .: {}
                f:image: {}
                f:imagePullPolicy: {}
                f:name: {}
                f:resources: {}
                f:securityContext:
                  .: {}
                  f:allowPrivilegeEscalation: {}
                  f:capabilities: {}
                  f:privileged: {}
                  f:readOnlyRootFilesystem: {}
                  f:runAsNonRoot: {}
                f:stdin: {}
                f:terminationMessagePath: {}
                f:terminationMessagePolicy: {}
                f:tty: {}
            f:dnsPolicy: {}
            f:imagePullSecrets:
              .: {}
              k:{"name":"dockerhub"}:
                .: {}
                f:name: {}
            f:restartPolicy: {}
            f:schedulerName: {}
            f:securityContext: {}
            f:terminationGracePeriodSeconds: {}
    manager: rancher
    operation: Update
    time: "2022-01-20T22:35:09Z"
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:deployment.kubernetes.io/revision: {}
      f:status:
        f:availableReplicas: {}
        f:conditions:
          .: {}
          k:{"type":"Available"}:
            .: {}
            f:lastTransitionTime: {}
            f:lastUpdateTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"Progressing"}:
            .: {}
            f:lastTransitionTime: {}
            f:lastUpdateTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
        f:observedGeneration: {}
        f:readyReplicas: {}
        f:replicas: {}
        f:updatedReplicas: {}
    manager: kube-controller-manager
    operation: Update
    time: "2022-01-20T22:35:12Z"
  name: debian
  namespace: obchod-uat
  resourceVersion: "56787224"
  uid: 18e4e476-05d2-4ab8-82a2-6faccf5d0e32
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      workload.user.cattle.io/workloadselector: deployment-obchod-uat-debian
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        cattle.io/timestamp: "2022-01-20T22:35:08Z"
      creationTimestamp: null
      labels:
        workload.user.cattle.io/workloadselector: deployment-obchod-uat-debian
    spec:
      containers:
      - image: debian:stretch
        imagePullPolicy: Always
        name: debian
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities: {}
          privileged: false
          readOnlyRootFilesystem: false
          runAsNonRoot: false
        stdin: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        tty: true
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: dockerhub
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-01-20T22:35:12Z"
    lastUpdateTime: "2022-01-20T22:35:12Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-01-20T22:35:09Z"
    lastUpdateTime: "2022-01-20T22:35:12Z"
    message: ReplicaSet "debian-6d9b7dbd46" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

output apt-get update (Debian):

Ign:1 http://security.debian.org/debian-security stretch/updates InRelease
Ign:2 http://security.debian.org/debian-security stretch/updates Release
Ign:3 http://security.debian.org/debian-security stretch/updates/main all Packages
Ign:4 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
Ign:3 http://security.debian.org/debian-security stretch/updates/main all Packages
Ign:4 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
Ign:3 http://security.debian.org/debian-security stretch/updates/main all Packages
Ign:4 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
Ign:3 http://security.debian.org/debian-security stretch/updates/main all Packages
Ign:4 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
Ign:3 http://security.debian.org/debian-security stretch/updates/main all Packages
Ign:4 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
Ign:3 http://security.debian.org/debian-security stretch/updates/main all Packages
Err:4 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
  403  Forbidden
Ign:5 http://deb.debian.org/debian stretch InRelease
Ign:6 http://deb.debian.org/debian stretch-updates InRelease
Ign:7 http://deb.debian.org/debian stretch Release
Ign:8 http://deb.debian.org/debian stretch-updates Release
Ign:9 http://deb.debian.org/debian stretch/main amd64 Packages
Ign:10 http://deb.debian.org/debian stretch/main all Packages
Ign:11 http://deb.debian.org/debian stretch-updates/main amd64 Packages
Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
Ign:9 http://deb.debian.org/debian stretch/main amd64 Packages
Ign:10 http://deb.debian.org/debian stretch/main all Packages
Ign:11 http://deb.debian.org/debian stretch-updates/main amd64 Packages
Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
Ign:9 http://deb.debian.org/debian stretch/main amd64 Packages
Ign:10 http://deb.debian.org/debian stretch/main all Packages
Ign:11 http://deb.debian.org/debian stretch-updates/main amd64 Packages
Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
Ign:9 http://deb.debian.org/debian stretch/main amd64 Packages
Ign:10 http://deb.debian.org/debian stretch/main all Packages
Ign:11 http://deb.debian.org/debian stretch-updates/main amd64 Packages
Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
Ign:9 http://deb.debian.org/debian stretch/main amd64 Packages
Ign:10 http://deb.debian.org/debian stretch/main all Packages
Ign:11 http://deb.debian.org/debian stretch-updates/main amd64 Packages
Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
Err:9 http://deb.debian.org/debian stretch/main amd64 Packages
  403  Forbidden
Ign:10 http://deb.debian.org/debian stretch/main all Packages
Err:11 http://deb.debian.org/debian stretch-updates/main amd64 Packages
  403  Forbidden
Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages
Reading package lists... Done
W: The repository 'http://security.debian.org/debian-security stretch/updates Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: The repository 'http://deb.debian.org/debian stretch Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: The repository 'http://deb.debian.org/debian stretch-updates Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/main/binary-amd64/Packages  403  Forbidden
E: Failed to fetch http://deb.debian.org/debian/dists/stretch/main/binary-amd64/Packages  403  Forbidden
E: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/main/binary-amd64/Packages  403  Forbidden
E: Some index files failed to download. They have been ignored, or old ones used instead.

It looks like all traffic goes to 85.93.165.117

root@debian-5cfb4cd49d-gj6qx:/# ping deb.debian.org
PING profiwh.com (85.93.165.117) 56(84) bytes of data.
64 bytes from web18.profiwh.com (85.93.165.117): icmp_seq=1 ttl=62 time=0.460 ms
64 bytes from web18.profiwh.com (85.93.165.117): icmp_seq=2 ttl=62 time=0.590 ms
64 bytes from web18.profiwh.com (85.93.165.117): icmp_seq=3 ttl=62 time=0.557 ms

Answers

Chosen as BEST ANSWER
- TomHapl
- January 21, 2022 at 8:52 pm
- 0 votes
0
I found this resolv.conf in node
```
search profiwh.cz mediaface.cz
nameserver 85.93.165.6
nameserver 85.93.165.4
```
in container was:
```
search obchod-uat.svc.cluster.local svc.cluster.local cluster.local profiwh.cz mediaface.cz
nameserver 10.43.0.10
options ndots:5
```
the debian container contains nsswitch.conf and busybox container not.

Not sure how nsswitch resolves dns but I removed search profiwh.cz mediaface.cz from nodes resolv.conf and it is working for now.

(Edit)

- Eetae0x
- January 20, 2022 at 7:30 pm
- 0 votes
0
Did you check if there are multiple A records setup?

For example nytimes.com has 4 possible results:
```
❯ dig +short nytimes.com
151.101.193.164
151.101.129.164
151.101.65.164
151.101.1.164
```
Login or Signup to reply.

Please signup or login to give your own answer.

Click here to cancel reply.

Different DNS resolution inside cluster – Debian

Answers