Routing to different services in same EKS using Application Load balancer based on paths - Nginx

Nandha
July 11, 2022
207 views
1 vote
2 Answers

Let me give brief about the question. We have deployed our flask services in EKS. Previously we were able to route to different service in the same EKS based on path using classic load balancer in AWS. Since Classic LB is deprecated, we migrated to Application LB. Here we are not able to do path based routing for flask services. How to achieve path based routing in ALB for flask service running in different port in same EKS cluster?

Detailed description is given below.

aws eks --region eu-west-2 update-kubeconfig --name alb-check
kubectl create namespace check

After that we have installed nginx

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install nginx-ingress ingress-nginx/ingress-nginx 
    --namespace check 
    --set controller.replicaCount=2 
    --set controller.nodeSelector."beta.kubernetes.io/os"=linux 
    --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

After running the above command, a classic ingress controller is created in the region with some DNS name.
Then we have deployed 2 similar services using following yaml.

#####
apiVersion: apps/v1
kind: Deployment
metadata:
  name: console
  namespace: check
  labels:
    app: console
spec:
  replicas: 1
  selector:
    matchLabels:
      app: console
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: console
    spec:
      containers:
      - name: console
        imagePullPolicy: Always
        image: xxxxxx.dkr.ecr.eu-west-1.amazonaws.com/check:sample_1

---
---

apiVersion: v1
kind: Service
metadata:
  name: console
  namespace: check
spec:
  selector:
    app: console
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 3000
  type: ClusterIP

The content of the docker image also given below.

from flask import Flask
from flask import jsonify

app_name = "Check1"
app = Flask(app_name)

portVal =  3000

@app.route('/', methods=['GET'])
def api0():
    data = {"code": 200, "source": "api1", "message": "This service 1", "port": portVal}
    return jsonify(data)


@app.route('/api1', methods=['GET'])
def api1():
    data = {"code": 200, "source": "api1", "port": portVal}
    return jsonify(data)


@app.route('/api2', methods=['GET'])
def api2():
    data = {"code": 200, "source": "api2", "port": portVal}
    return jsonify(data)


if __name__ == "__main__":
    app.run()

Corresponding docker file.

FROM python:3.7
RUN mkdir /check
ADD ./s1/app_file.py /check/app_file.py
ADD ./s1/s1_requirements.txt /check/s1_requirements.txt
WORKDIR /check
RUN pip install -r ./s1_requirements.txt
EXPOSE 3000

CMD ["gunicorn", "-b", "0.0.0.0:3000", "app_file:app", "--workers", "3", "--threads", "25", "--timeout", "660"]

The file s1_requirements.txt had only 2 requirements.

Flask==2.1.2
gunicorn==19.9.0

I have deployed another similar service only difference is it is running on different port 3001. After deploying both services, i have configured ingress using the following file.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: check-ingress
  namespace: check
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/cors-allow-headers: "Content-Type,authToken,sdk-Key,data,api-type,Authorization,userId,teamId"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE, OPTIONS"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/client-body-buffer-size: "16m"
    nginx.ingress.kubernetes.io/proxy-body-size: "16m"
    nginx.ingress.kubernetes.io/enable-modsecurity: "true"
spec:
  rules:
    - http:
        paths:
          - pathType: Prefix
            backend:
              service:
                name: console
                port:
                  number: 3000
            path: /console/(.*)

          - pathType: Prefix
            backend:
              service:
                name: console2
                port:
                  number: 3001
            path: /console2/(.*)

kubectl apply -f ingress.yaml

So far things works as expected.
when i give DNSURL/console/api1 => I am able to get response from service 1’s api1 which is running on port 3000.

Similarly
when i give DNSURL/console2/api1 => I am able to get response from service 2’s api1 which is running on port 3001.

Since classic load balancer is deprecated we thought of using Application load balancer for the above use case.
There we are not able to achieve the routing to services running on different ports based on the paths.

Ran the following commands to create a new EKS cluster and to create ALB.
Reference aws doc

eksctl create cluster --name alb3-check --region eu-west-2 --version 1.22

eksctl utils associate-iam-oidc-provider --region eu-west-2 --cluster alb3-check --approve

aws eks describe-cluster --name alb3-check --query "cluster.identity.oidc.issuer" --region eu-west-2 --output text

aws iam create-policy --policy-name alb3tqAWSLoadBalancerControllerIAMPolicy --policy-document file://iam-policy.json


eksctl create iamserviceaccount 
--cluster=alb3-check 
--namespace=kube-system 
--name=aws-load-balancer-controller 
--attach-policy-arn=arn:aws:iam::xxxxxxxxx74:policy/alb3tqAWSLoadBalancerControllerIAMPolicy 
--override-existing-serviceaccounts 
--approve

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml

curl -Lo v2_4_1_full.yaml https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/download/v2.4.1/v2_4_1_full.yaml

sed -i.bak -e 's|your-cluster-name|alb3-check|' ./v2_4_1_full.yaml

kubectl apply -f v2_4_1_full.yaml (Run this twice)

After creating ALB, i have deployed same services in this EKS.

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: check
  name: console
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: console
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: console
    spec:
      containers:
        - name: console
          imagePullPolicy: Always
          image: "xxxxxx.dkr.ecr.eu-west-1.amazonaws.com/check:sample_1"
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: console
  namespace: check
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: console
  ports:
  - protocol: TCP
    port: 3000
    targetPort: 3000

Deployed second service which is running on different port 3001.
ingress.yaml content is given below

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: check
  name: ingress-check
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/load-balancer-name: t3
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/tags: product=check
    alb.ingress.kubernetes.io/group.name: test
spec:
  rules:
    - host: t3-xxxxxx.eu-west-1.elb.amazonaws.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: console
                port:
                  number: 3000

When i give path as ‘/’ in the above yaml, things are working for service 1. I am able to access URL/api or
URL/api2

When i give path as ‘/console’ and apply the yaml,
i am unable to access the APIs (in classic load balancer we were able to do it).
URL/console/api ,
URL/console/api2 are not accessible.

Due to this path, issue we are not able to route to multiple services running on different ports based on paths.

Answers

Chosen as BEST ANSWER

Thanks to @Binh Nguyen's answer, we came to know that URL rewrite is not supported on ALB. So currently we have a workaround and i am posting that here.

Since the path given is not removed during redirect, we have added the same path as url prefix in blue print for the flask services.

from flask import Flask, Blueprint
from flask import jsonify

app_name = "Check1"
app_blueprint = Blueprint('app', __name__)
app = Flask(app_name)

curr_port = 3000


@app_blueprint.route('/', methods=['GET'])
def rootapi0():
    data = {"code": 200, "source": "api1", "message": "This root service 1", "port": curr_port}
    return jsonify(data)


@app_blueprint.route('/api1', methods=['GET'])
def api1():
    data = {"code": 200, "source": "api1", "port": curr_port}
    return jsonify(data)


@app_blueprint.route('/api2', methods=['GET'])
def api2():
    data = {"code": 200, "source": "api2", "port": curr_port}
    return jsonify(data)


app.register_blueprint(app_blueprint, url_prefix='/console')

if __name__ == "__main__":
    app.run()

Notice that we have introduced blue print for api routes and url_prefix='/console' for the blue print.

In the same way, we have changed the second flask service with url_prefix as /console2

After building and deploying this service, for routing we have given following ingress.yaml.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: check
  name: ingress-check
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/load-balancer-name: t5
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/tags: product=check
    alb.ingress.kubernetes.io/group.name: test
spec:
  rules:
    - http:
        paths:
          - path: /console
            pathType: Prefix
            backend:
              service:
                name: console
                port:
                  number: 3000   
          - path: /console2
            pathType: Prefix
            backend:
              service:
                name: console2
                port:
                  number: 3001

With the above yaml file, we are able to redirect to corresponding services.

(Edit)

- BinhNguyen
- July 11, 2022 at 4:42 pm
- 0 votes
0
I think what you would like to implement is the URL rewrite capability of AWS ALB Controller. Unfortunately, this feature is not supported by now.

You can check this out at:
- https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/835
There are also mentioned work-arounds that you can apply, such as an additional backend to redirect your application to your preferred path.
```
    rules:
      - host: main-app.net
        http: 
          paths:
            - path: /app
              pathType: Prefix
              backend:
                service:
                  name: main-app
                  port: 
                    number: 8080
            - path: /
              pathType: Exact
              backend:
                service:
                  name: redirect-to-main-app
                  port: 
                    name: use-annotation
```
or you can consider another approach by using different Host for your application. For example:
- console1.app.net
- console2.app.net
Login or Signup to reply.

Please signup or login to give your own answer.

Click here to cancel reply.

Routing to different services in same EKS using Application Load balancer based on paths – Nginx

Answers