I am trying to reach an application on an EKS cluster via an internal NLB. The request is timing out.
- the EKS is in a non-routable cidr (10.0.0.0/16)
- the NLB is in the routable 10.1.0.0/16 (from on-prem) cidr and it forwards the traffic to the backend via the Istio Ingress Gateway
- the on-prem cidr is something like 10.2.0.0/16 and it can talk to 10.1.0.0/16 where the NLB is located (routable cidr)
- the application is reachable from inside the VPC (via the same NLB)
- the application is not reachable from on-prem (via the same NLB) – timing out
- I can hit another application (not EKS) via an internal ALB in the same routable cidr (same port 443) with the backend in the non-routable cidr (hence proving the connectivity)
- I can also reach VPC endpoints in the same VPC routable cidr
- the connection from on-prem to AWS VPC cidr is going through a Direct Connect connection
- there is no access to the internet
I am thinking that perhaps how the NLB forwards the traffic might have an impact on this. DNS is resolving to the NLB IPs but won’t connect. I can’t telnet to the NLB IPs either though.
Since EKS is in 10.0.0.0/16, it can talk to 10.1.0.0/16 which is routed locally in the vpc but it can’t talk to on-prem 10.2.0.0/16. Then the NLB does not terminate the connection but lets the traffic through with the source IP, hence that IP won’t be allowed in the EKS cidr.
Any pointers are highly appreciated.
2
Answers
In my case the on-prem CIDR is not associated with the EKS CIDR. Hence the source IP won't be allowed in the EKS CIDR. But that is what the NLB will do. Hence I had to make sure that that is not enabled.
Solution: The "Preserve Client IP" option in the target group should be set to DISABLED.
The NLB’s are transparent and don’t have any security group, check if the application hosted on EKS has the on prem CIDR range allowed for that port. This security group could be attached to the EKS pod or to the EKS worker node group.