Question posted in 
Service definition
apiVersion: v1
kind: Service
metadata:
name: zevrant-oauth2-service-db
spec:
ports:
- port: 5432
targetPort: 5432
selector:
app: zevrant-oauth2-service-db
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: zevrant-oauth2-service-db-deployment
labels:
app: zevrant-oauth2-service-db
spec:
replicas: 1
selector:
matchLabels:
app: zevrant-oauth2-service-db
template:
metadata:
labels:
app: zevrant-oauth2-service-db
spec:
imagePullSecrets:
- name: regcred
volumes:
- name: database
nfs:
server: 192.168.0.127
path: /i-data/420aa917/nfs/keys
nodeSelector:
architecture: arm
containers:
- name: zevrant-oauth2-service-db
image: postgres:latest
volumeMounts:
- name: database
mountPath: /storage/keys/
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: oauth-db-password
key: password
- name: PGDATA
value: /storage/keys/db/$ENVIRONMENT/oauth2/
- name: POSTGRES_USER
value: zevrant
- name: POSTGRES_DB
value: oauth2
ports:
- containerPort: 5432
cluster detail information
zevrant@master-node:~$ kubectl get svc -o wide -n kube-system|grep dns
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 10h k8s-app=kube-dns
zevrant@master-node:~$ kubectl get pod -o wide -n kube-system|grep dns
coredns-66bff467f8-nq5jv 1/1 Running 1 150m 10.32.0.7 zevrant <none> <none>
coredns-66bff467f8-tljmr 1/1 Running 1 10h 10.40.0.1 master-node <none> <none>
kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
ingress-nodeport-develop NodePort 10.99.251.215 <none> 8080:30124/TCP 10h app=zevrant-home-ui
zevrant-oauth2-service-db ClusterIP 10.97.75.171 <none> 5432/TCP 9h app=zevrant-oauth2-service-db
coredns logs for both master and worker node consist of
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b
query for destination service
zevrant-home-ui@zevrant-home-ui-deployment-79c7dbb78-gksn8:/$ dig @10.96.0.10 zevrant-oauth2-service-db
; <<>> DiG 9.11.3-1ubuntu1.12-Ubuntu <<>> @10.96.0.10 zevrant-oauth2-service-db
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65289
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;zevrant-oauth2-service-db. IN A
;; ANSWER SECTION:
zevrant-oauth2-service-db. 5 IN A 92.242.140.2
;; Query time: 14 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Sun Jul 12 23:43:07 UTC 2020
;; MSG SIZE rcvd: 95
/etc/resolve.conf
nameserver 10.96.0.10
search zevrant-home-services-develop.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
destination pod can hit the service (which is forwarded to itself) but UI pods cannot establish a connection to the service or destination pod via IP address. Any thoughts as to why this might be?
EDIT #1
postgress logs
2020-07-13 15:34:27.394 UTC [1] LOG: starting PostgreSQL 12.3 (Debian 12.3-1.pgdg100+1) on arm-unknown-linux-gnueabihf, compiled by gcc (Debian 8.3.0-6) 8.3.0, 32-bit
2020-07-13 15:34:27.395 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-07-13 15:34:27.395 UTC [1] LOG: listening on IPv6 address "::", port 5432
2020-07-13 15:34:28.825 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-07-13 15:34:29.401 UTC [26] LOG: database system was shut down at 2020-07-13 15:31:07 UTC
2020-07-13 15:34:29.508 UTC [1] LOG: database system is ready to accept connections
weave report
{
"Ready": true,
"Version": "2.6.5",
"VersionCheck": {
"Enabled": true,
"Success": false,
"NewVersion": "",
"NextCheckAt": "2020-07-13T19:06:21.030492043Z"
},
"Router": {
"Protocol": "weave",
"ProtocolMinVersion": 1,
"ProtocolMaxVersion": 2,
"Encryption": false,
"PeerDiscovery": true,
"Name": "ea:99:0e:11:d5:ae",
"NickName": "master-node",
"Port": 6783,
"Peers": [
{
"Name": "ea:99:0e:11:d5:ae",
"NickName": "master-node",
"UID": 4593396642225438236,
"ShortID": 2254,
"Version": 4,
"Connections": [
{
"Name": "1a:21:59:47:9a:5e",
"NickName": "zevrant",
"Address": "192.168.0.207:38089",
"Outbound": false,
"Established": true
},
{
"Name": "6a:6f:c4:0b:db:46",
"NickName": "raspi-03-02",
"Address": "192.168.0.42:6783",
"Outbound": true,
"Established": true
}
]
},
{
"Name": "6a:6f:c4:0b:db:46",
"NickName": "raspi-03-02",
"UID": 13536333655574378111,
"ShortID": 3959,
"Version": 13,
"Connections": [
{
"Name": "1a:21:59:47:9a:5e",
"NickName": "zevrant",
"Address": "192.168.0.207:45287",
"Outbound": false,
"Established": true
},
{
"Name": "ea:99:0e:11:d5:ae",
"NickName": "master-node",
"Address": "192.168.0.73:50257",
"Outbound": false,
"Established": true
}
]
},
{
"Name": "1a:21:59:47:9a:5e",
"NickName": "zevrant",
"UID": 14270350061285030667,
"ShortID": 3515,
"Version": 4,
"Connections": [
{
"Name": "6a:6f:c4:0b:db:46",
"NickName": "raspi-03-02",
"Address": "192.168.0.42:6783",
"Outbound": true,
"Established": true
},
{
"Name": "ea:99:0e:11:d5:ae",
"NickName": "master-node",
"Address": "192.168.0.73:6783",
"Outbound": true,
"Established": true
}
]
}
],
"UnicastRoutes": [
{
"Dest": "6a:6f:c4:0b:db:46",
"Via": "6a:6f:c4:0b:db:46"
},
{
"Dest": "1a:21:59:47:9a:5e",
"Via": "1a:21:59:47:9a:5e"
},
{
"Dest": "ea:99:0e:11:d5:ae",
"Via": "00:00:00:00:00:00"
}
],
"BroadcastRoutes": [
{
"Source": "ea:99:0e:11:d5:ae",
"Via": [
"6a:6f:c4:0b:db:46",
"1a:21:59:47:9a:5e"
]
},
{
"Source": "1a:21:59:47:9a:5e",
"Via": null
},
{
"Source": "6a:6f:c4:0b:db:46",
"Via": null
}
],
"Connections": [
{
"Address": "192.168.0.42:6783",
"Outbound": true,
"State": "established",
"Info": "fastdp 6a:6f:c4:0b:db:46(raspi-03-02)",
"Attrs": {
"mtu": 1376,
"name": "fastdp"
}
},
{
"Address": "192.168.0.207:38089",
"Outbound": false,
"State": "established",
"Info": "fastdp 1a:21:59:47:9a:5e(zevrant)",
"Attrs": {
"mtu": 1376,
"name": "fastdp"
}
},
{
"Address": "192.168.0.100:6783",
"Outbound": true,
"State": "failed",
"Info": "dial tcp :0-u003e192.168.0.100:6783: connect: connection refused, retry: 2020-07-13 15:53:46.887472731 +0000 UTC m=+60329.814440915",
"Attrs": null
},
{
"Address": "192.168.0.82:6783",
"Outbound": true,
"State": "failed",
"Info": "dial tcp :0-u003e192.168.0.82:6783: connect: connection refused, retry: 2020-07-13 15:47:56.145112536 +0000 UTC m=+59979.072080717",
"Attrs": null
}
],
"TerminationCount": 0,
"Targets": [
"192.168.0.82",
"192.168.0.42",
"192.168.0.100",
"192.168.0.207"
],
"OverlayDiagnostics": {
"fastdp": {
"Vports": [
{
"ID": 0,
"Name": "datapath",
"TypeName": "internal"
},
{
"ID": 1,
"Name": "vethwe-datapath",
"TypeName": "netdev"
},
{
"ID": 2,
"Name": "vxlan-6784",
"TypeName": "vxlan"
}
],
"Flows": [
{
"FlowKeys": [
"EthernetFlowKey{src: ca:03:86:42:44:3b, dst: 1a:21:59:47:9a:5e}",
"InPortFlowKey{vport: 1}",
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
],
"Actions": [
"SetTunnelAction{id: 0000000000dbb8ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.207, ttl: 64, df: true}",
"OutputAction{vport: 2}"
],
"Packets": 24,
"Bytes": 4230,
"Used": 59955261
},
{
"FlowKeys": [
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
"EthernetFlowKey{src: ea:99:0e:11:d5:ae, dst: 01:00:5e:7f:ff:fa}",
"InPortFlowKey{vport: 1}",
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
],
"Actions": [
"SetTunnelAction{id: 0000000000f778ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.42, ttl: 64, df: true}",
"OutputAction{vport: 2}",
"SetTunnelAction{id: 0000000000dbb8ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.207, ttl: 64, df: true}",
"OutputAction{vport: 2}",
"OutputAction{vport: 0}"
],
"Packets": 40,
"Bytes": 12240,
"Used": 60041269
},
{
"FlowKeys": [
"TunnelFlowKey{id: 00000000008cedbb, ipv4src: 192.168.0.207, ipv4dst: 192.168.0.73}",
"InPortFlowKey{vport: 2}",
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
"EthernetFlowKey{src: 1a:21:59:47:9a:5e, dst: ca:03:86:42:44:3b}",
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}"
],
"Actions": [
"OutputAction{vport: 1}"
],
"Packets": 24,
"Bytes": 2106,
"Used": 59955261
},
{
"FlowKeys": [
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
"EthernetFlowKey{src: ee:7c:a4:ba:4d:12, dst: ca:03:86:42:44:3b}",
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
"InPortFlowKey{vport: 2}",
"TunnelFlowKey{id: 00000000008cedbb, ipv4src: 192.168.0.207, ipv4dst: 192.168.0.73}",
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}"
],
"Actions": [
"OutputAction{vport: 1}"
],
"Packets": 5,
"Bytes": 414,
"Used": 59953149
},
{
"FlowKeys": [
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
"EthernetFlowKey{src: 42:8b:89:88:2b:c3, dst: ca:03:86:42:44:3b}",
"InPortFlowKey{vport: 2}",
"TunnelFlowKey{id: 00000000008cedbb, ipv4src: 192.168.0.207, ipv4dst: 192.168.0.73}"
],
"Actions": [
"OutputAction{vport: 1}"
],
"Packets": 4,
"Bytes": 398,
"Used": 59942269
},
{
"FlowKeys": [
"EthernetFlowKey{src: ca:03:86:42:44:3b, dst: ee:7c:a4:ba:4d:12}",
"InPortFlowKey{vport: 1}",
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
],
"Actions": [
"SetTunnelAction{id: 0000000000dbb8ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.207, ttl: 64, df: true}",
"OutputAction{vport: 2}"
],
"Packets": 5,
"Bytes": 1032,
"Used": 59953149
},
{
"FlowKeys": [
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
"EthernetFlowKey{src: ca:03:86:42:44:3b, dst: 8a:0a:d2:ae:d3:97}",
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
"InPortFlowKey{vport: 1}",
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}"
],
"Actions": [
"SetTunnelAction{id: 0000000000dbb8ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.207, ttl: 64, df: true}",
"OutputAction{vport: 2}"
],
"Packets": 6,
"Bytes": 1592,
"Used": 59954109
},
{
"FlowKeys": [
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
"TunnelFlowKey{id: 00000000008cedbb, ipv4src: 192.168.0.207, ipv4dst: 192.168.0.73}",
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
"EthernetFlowKey{src: 8a:0a:d2:ae:d3:97, dst: ca:03:86:42:44:3b}",
"InPortFlowKey{vport: 2}",
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
],
"Actions": [
"OutputAction{vport: 1}"
],
"Packets": 6,
"Bytes": 542,
"Used": 59954109
},
{
"FlowKeys": [
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
"EthernetFlowKey{src: be:ae:7d:4e:72:5a, dst: ff:ff:ff:ff:ff:ff}",
"TunnelFlowKey{id: 00000000008cef77, ipv4src: 192.168.0.42, ipv4dst: 192.168.0.73}",
"InPortFlowKey{vport: 2}",
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}"
],
"Actions": [
"OutputAction{vport: 1}",
"OutputAction{vport: 0}"
],
"Packets": 10,
"Bytes": 1784,
"Used": 60044189
},
{
"FlowKeys": [
"InPortFlowKey{vport: 1}",
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
"EthernetFlowKey{src: ca:03:86:42:44:3b, dst: 42:8b:89:88:2b:c3}",
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}"
],
"Actions": [
"SetTunnelAction{id: 0000000000dbb8ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.207, ttl: 64, df: true}",
"OutputAction{vport: 2}"
],
"Packets": 4,
"Bytes": 1016,
"Used": 59942269
},
{
"FlowKeys": [
"UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
"InPortFlowKey{vport: 2}",
"EthernetFlowKey{src: 3a:96:00:f9:20:d1, dst: ff:ff:ff:ff:ff:ff}",
"TunnelFlowKey{id: 00000000008cef77, ipv4src: 192.168.0.42, ipv4dst: 192.168.0.73}",
"UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
"UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
"UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
],
"Actions": [
"OutputAction{vport: 1}",
"OutputAction{vport: 0}"
],
"Packets": 3,
"Bytes": 1149,
"Used": 59998761
}
]
},
"sleeve": null
},
"TrustedSubnets": [],
"Interface": "datapath (via ODP)",
"CaptureStats": {
"FlowMisses": 12234
},
"MACs": [
{
"Mac": "ee:7c:a4:ba:4d:12",
"Name": "1a:21:59:47:9a:5e",
"NickName": "zevrant",
"LastSeen": "2020-07-13T15:43:17.292214531Z"
},
{
"Mac": "ca:03:86:42:44:3b",
"Name": "ea:99:0e:11:d5:ae",
"NickName": "master-node",
"LastSeen": "2020-07-13T15:43:17.292142775Z"
},
{
"Mac": "ea:99:0e:11:d5:ae",
"Name": "ea:99:0e:11:d5:ae",
"NickName": "master-node",
"LastSeen": "2020-07-13T15:43:17.292177972Z"
},
{
"Mac": "8a:0a:d2:ae:d3:97",
"Name": "1a:21:59:47:9a:5e",
"NickName": "zevrant",
"LastSeen": "2020-07-13T15:43:17.292261033Z"
},
{
"Mac": "3a:96:00:f9:20:d1",
"Name": "6a:6f:c4:0b:db:46",
"NickName": "raspi-03-02",
"LastSeen": "2020-07-13T15:43:17.292295962Z"
},
{
"Mac": "be:ae:7d:4e:72:5a",
"Name": "6a:6f:c4:0b:db:46",
"NickName": "raspi-03-02",
"LastSeen": "2020-07-13T15:43:17.292272079Z"
},
{
"Mac": "1a:21:59:47:9a:5e",
"Name": "1a:21:59:47:9a:5e",
"NickName": "zevrant",
"LastSeen": "2020-07-13T15:43:17.292200309Z"
},
{
"Mac": "42:8b:89:88:2b:c3",
"Name": "1a:21:59:47:9a:5e",
"NickName": "zevrant",
"LastSeen": "2020-07-13T15:43:17.292226062Z"
}
]
},
"IPAM": {
"Paxos": null,
"Range": "10.32.0.0/12",
"RangeNumIPs": 1048576,
"ActiveIPs": 2,
"DefaultSubnet": "10.32.0.0/12",
"Entries": [
{
"Token": "10.32.0.0",
"Size": 393216,
"Peer": "1a:21:59:47:9a:5e",
"Nickname": "zevrant",
"IsKnownPeer": true,
"Version": 75
},
{
"Token": "10.38.0.0",
"Size": 131072,
"Peer": "6a:6f:c4:0b:db:46",
"Nickname": "raspi-03-02",
"IsKnownPeer": true,
"Version": 15
},
{
"Token": "10.40.0.0",
"Size": 262144,
"Peer": "ea:99:0e:11:d5:ae",
"Nickname": "master-node",
"IsKnownPeer": true,
"Version": 6
},
{
"Token": "10.44.0.0",
"Size": 1,
"Peer": "6a:6f:c4:0b:db:46",
"Nickname": "raspi-03-02",
"IsKnownPeer": true,
"Version": 1
},
{
"Token": "10.44.0.1",
"Size": 262143,
"Peer": "ea:99:0e:11:d5:ae",
"Nickname": "master-node",
"IsKnownPeer": true,
"Version": 0
}
],
"PendingClaims": null,
"PendingAllocates": null
}
}
Pods are deployed into the same namespace as such
NAME READY STATUS RESTARTS AGE
zevrant-home-ui-deployment-79c7dbb78-gksn8 1/1 Running 0 16h
zevrant-home-ui-deployment-79c7dbb78-zgp4c 1/1 Running 0 16h
zevrant-oauth2-service-db-deployment-6f7c7ccdb5-5g599 1/1 Running
0 19m
2
Answers
The solution ended up being to run the following commands on the master and all slave nodes. This is needed any time there are changes to the networking settings on the cluster.
Based on the
digoutput you sharedzevrant-oauth2-service-dbis resolving to92.242.140.2but it looks like the IP address of your K8s service is10.97.75.171(ClusterIP) (based on the output you shared too).If you hit
10.97.75.171 5432you should be able to access your Postgres database, provided that you don’t have any Kubernetes Network Policy and/or firewall blocking access. Make sure you that in your Postgres config you are binding the server to0.0.0.0otherwise if it’s something likelocalhostyou will only be able to get to it from the pod.So the question is what is
92.242.140.2? Wny is coredns responding to a query tozevrant-oauth2-service-dbwith92.242.140.2? Is there a DNS forwarder configured in coredns? Is there a default domain configured that is not part ofsvc.cluster.local?