I am trying to set up an Express.js application running in a Kubernetes pod to connect to a MongoDB Atlas database. I’ve set up an egress droplet to control and log the outbound traffic from the cluster. The IP of the egress droplet has been whitelisted in MongoDB Atlas, but the application still can’t connect to the database. I’m looking for guidance on how to correctly configure the Express app, or the Kubernetes cluster, to route traffic through the egress droplet.
Environment Details:
Express App: Running in a Kubernetes pod.
Egress Droplet: Set up to control outbound traffic from the Kubernetes cluster.
Database: MongoDB Atlas, with the egress droplet’s IP whitelisted.
Current Configuration:
- Network Policy: I have a network policy in place that’s intended to allow egress traffic from the pod to the egress droplet.
Here is my allow-egress-via-gateway.yaml
that is applied:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-egress-via-gateway
spec:
podSelector:
matchExpressions:
- {
key: app,
operator: In,
values: [
express-app
]
}
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 153.000.000.000/32 # The egress gateway IP
# Additional rules for DNS traffic
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
Here is my express-deployment.yaml
, which creates the express-app pod:
apiVersion: apps/v1
kind: Deployment
metadata:
name: express-app-deployment
spec:
replicas: 1
selector:
matchLabels:
app: express-app
template:
metadata:
labels:
app: express-app
annotations:
kubectl.kubernetes.io/restartedAt: "$(date +%s)"
spec:
containers:
- name: express-app
image: mydockerhub/myexpress-backend:latest
ports:
- containerPort: 8000
envFrom:
- secretRef:
name: express-env-secret
imagePullSecrets:
- name: my-docker-credentials
- IP Masquerading: On the egress droplet, I’ve set up IP masquerading rules in iptables to modify the source IP of the outgoing packets.
- Database Connection: The Express app is configured to connect to MongoDB Atlas using a connection string provided by Atlas.
Issues Encountered:
Despite the above setup, the Express app fails to connect to MongoDB Atlas. The logs suggest it’s a connection issue, likely related to the IP not being whitelisted, even though I’ve confirmed the egress droplet’s IP is on the whitelist.
Specific Questions:
- How should I configure the Express app or the Kubernetes pod to ensure it routes traffic through the egress droplet?
- Are there any common pitfalls or additional steps I need to consider when setting up an egress controller for a Kubernetes cluster?
- Is there any additional configuration required on the egress droplet to ensure it properly routes and logs the traffic from the Kubernetes pods?
Any insights, suggestions, or further diagnostic steps would be greatly appreciated!
2
Answers
Make sure your network policy (
allow-egress-via-gateway.yaml
) correctly allows traffic to the egress droplet. The policy you provided looks correct, but double-check the egress droplet IP (the cidr part:cidr: <Egress-Droplet-IP>/32
).Make sure the deployment (
kubectl apply -f express-deployment.yaml
) is successful and the pod is running.On the egress droplet, you might use a command like the following to set up IP masquerading:
Confirm that IP masquerading is correctly set up on the egress droplet. That step is very important for ensuring that responses from MongoDB Atlas are routed back to the correct pod.
So check if the IP masquerade rule is present in the
iptables
configuration:sudo iptables -t nat -L POSTROUTING
, look for a line similar to:MASQUERADE all -- anywhere anywhere
Verify that the egress droplet correctly routes traffic to MongoDB Atlas and logs this traffic. Make sure no firewall or security rules on the egress droplet are blocking traffic to MongoDB Atlas.
In your Express app, use environment variables to store the MongoDB connection string securely:
Make sure the Express app uses the correct MongoDB Atlas connection string. That connection string should be stored securely, for example, in a Kubernetes Secret.
Verify that the necessary environment variables are correctly passed into the Express app via Kubernetes secrets or config maps.
Make sure to include error handling in your Express app for MongoDB connection, like:
You can also implement retry logic in your Express app, in case of transient network issues:
After all that, test the connection from within your pod (from the Express app to MongoDB Atlas)
And check the pod’s logs:
kubectl logs <pod-name>
Inside the pod, use network tools for diagnosing:
(more cently,
tracepath
ormtr
)Based on the documentation, you should be able to connect your express app with the database by retrieving secrets of the MongoDB Atlas and creating secrets, see sample below: