Summary of Setup
I have a remote Openshift Cluster with three pods. An nginx pod that serves a web app, a .NET pod that serves a .NET web api, and a Postgres database pod.
Problem
I am able to connect the nginx pod to the .NET pod and have no problem making api request. I cannot however get communication from the .NET pod to the Postgres pod in the Openshift cluster. I can curl the Postgres pod from the .NET pod’s terminal in Openshift web console and am able to connect the Postgres pod itself (not the database) using the Postrgres pod’s service name, so DNS resolution of the Postgres pod is working. Using Openshift’s port forwarding to forward traffic from my local machine’s localhost:5432 to my Postgres Pod’s port 5432, I can connect to the Postgres datbase while running the .NET API locally using the connection string Host=localhost;Port=5432;Database=postgres;Username=postgres;Password=postgres
. I can query, insert, etc with no problem as long as my connection to the database local. So connecting to the database via localhost clearly works. But when inside the Openshift cluster, I can’t get the .NET pod to connect to Postgres Pod database. The database should see a connection from another pod as a remote connection right? But it isn’t working at all. The postgres logs don’t show anything about a connection attempt either.
My .NET uses Host=postgres;Port=5432;Database=postgres;Username=postgres;Password=postgres
. This is stored in my appsettings.json. This is read in during in my Startup.cs.
// (U) add database context
services.AddDbContext<StigContext>(
options => options.UseNpgsql(Configuration["DbConnectionString"])
);
.NET database connection method, this endpoint returns true when I run and connect to the remote Postgres pod locally on my machine via Openshift’s port forwarding.
public IActionResult TestDatabaseConnection()
{
try
{
stigContext.Database.OpenConnection();
return Ok(stigContext.Database.CanConnect());
}
catch (Exception e)
{
var error = e.Message + "n" + e.ToString() + "n" + e.StackTrace;
return Ok(error);
}
}
Postgres Deployment and Service, PVC is done through the Openshift web console
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
spec:
selector:
matchLabels:
app: postgres
replicas: 1
template:
metadata:
annotations:
sidecar.istio.io/inject: "true"
sidecar.istio.io/proxyCPULimit: 200m
sidecar.istio.io/proxyMemory: 64Mi
sidecar.istio.io/proxyMemoryLimit: 256Mi
sidecar.istio.io/rewriteAppHTTPProbers: "true"
labels:
app: postgres
version: v1
spec:
containers:
- name: postgres
image: custom-docker-image-for-postgres:latest
imagePullPolicy: Always
env:
- name: POSTGRES_DB
value: postgres
- name: POSTGRES_USER
value: postgres
- name: POSTGRES_PASSWORD
value: postgres
- name: POSTGRES_HOST_AUTH_METHOD
value: trust
- name: PGDATA
value: /var/lib/postgresql/data/mydata
ports:
- containerPort: 5432
volumeMounts:
- name: postgres-pv-storage
mountPath: /var/lib/postgresql/data
resources:
requests:
memory: "128Mi"
cpu: "30m"
limits:
memory: "512Mi"
cpu: "60m"
volumes:
- name: postgres-pv-storage
persistentVolumeClaim:
claimName: postgres-pv-claim
---
apiVersion: v1
kind: Service
metadata:
labels:
app: postgres
service: postgres
name: postgres
spec:
ports:
- name: http
port: 5432
selector:
app: postgres
I ran a few command on the running Postgres pod to show the running config…
$ postgres -C listen_addresses
*
$ postgres -C hba_file
/var/lib/postgresql/data/mydata/pg_hba.conf
$ cat /var/lib/postgresql/data/mydata/pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local replication all trust
host replication all 127.0.0.1/32 trust
host replication all ::1/128 trust
# warning trust is enabled for all connections
# see https://www.postgresql.org/docs/12/auth-trust.html
host all all all trust
$ postgres -C port
5432
It seems like it should be able to connect remotely via pod to pod communication but definitely does not. I am at a loss as to what the issue is. I have no idea why its not working. I can provide any additional info if needed. Hopefully this is clear. Any suggestions are appreciated.
2
Answers
tdlr: my service definition's port name for my postgres service was 'http' and not 'tcp'. The name field is actually is the protocol so it does matter what you put here.
solution found here https://github.com/istio/istio/issues/16506#issuecomment-636224246
This may not be an issue if your deployment does not use istio.
The updated service yml for my postgres deployment is
Change the Host to postgesql service dns (Postrgres pod’s service name)in appsettings.json.