Question 98011 in Nginx Question posted in
The official documentation can be found here.

How to build an ingress controller with central cert and auth management – Nginx

I want to expose a few webapps in EKS to the internet in a centrally managed secure way.

In AWS, using an ALB is nice, as it for example allows you to terminate TLS and add authentication using Cognito. (see here)

To provision an ALB and connect it to the application there is the aws-load-balancer-controller.
It works fine, but it requires for each and every app/ingress to configure a new ALB:

  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/tags: Environment=test,Project=cognito
    external-dns.alpha.kubernetes.io/hostname: sample.${COK_MY_DOMAIN} 
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/auth-type: cognito
    alb.ingress.kubernetes.io/auth-scope: openid
    alb.ingress.kubernetes.io/auth-session-timeout: '3600'
    alb.ingress.kubernetes.io/auth-session-cookie: AWSELBAuthSessionCookie
    alb.ingress.kubernetes.io/auth-on-unauthenticated-request: authenticate
    alb.ingress.kubernetes.io/auth-idp-cognito: '{"UserPoolArn": "$(aws cognito-idp describe-user-pool --user-pool-id $COK_COGNITO_USER_POOL_ID --region $COK_AWS_REGION --query 'UserPool.Arn' --output text)","UserPoolClientId":"${COK_COGNITO_USER_POOL_CLIENT_ID}","UserPoolDomain":"${COK_COGNITO_DOMAIN}.auth.${COK_AWS_REGION}.amazoncognito.com"}'
    alb.ingress.kubernetes.io/certificate-arn: $COK_ACM_CERT_ARN 
    alb.ingress.kubernetes.io/target-type: 'ip'

I would love to have one central well defined ALB and all the application do not need to care about this anymore.

My idea was having a regular nginx-ingress-controller and expose it via a central ALB.

Now the question is: How do I connect the ALB to the nginx-controller?

One way would be manually configuring the ALB and build the target group by hand, which does not feel like a stable solution.

Another way would be using aws-load-balancer-controller to connect the nginx. In that case however nginx seems not to be able to publish the correct loadbalancer address and external-dns will enter the wrong DNS records. (Unfortunately there seems to be no –publish-ingress option in usual ingress controllers like nginx or traefik.)

Question:

  • Is there a way to make the nginx-ingress-controller provide the correct address?
  • Is there maybe an easier way that combining two ingress controllers?
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I think I found a good solution.

    I set up my environment using terraform. After I set up the alb ingress controller, I can create a suitable ingress object, wait until the ALB is up, use terraform to extract the address of the ALB and use publish-status-address to tell nginx to publish exactly that address to all its ingresses:

    resource "kubernetes_ingress_v1" "alb" {
  wait_for_load_balancer = true
  metadata {
    name = "alb"
    namespace = "kube-system"
    annotations = {
      "alb.ingress.kubernetes.io/scheme" = "internet-facing"
      "alb.ingress.kubernetes.io/listen-ports" = "[{"HTTP": 80}, {"HTTPS":443}]"
      "alb.ingress.kubernetes.io/ssl-redirect" = "443"
      "alb.ingress.kubernetes.io/certificate-arn" = local.cert
      "alb.ingress.kubernetes.io/target-type" = "ip"
    }
  }
  spec {
    ingress_class_name = "alb"
    default_backend {
      service {
        name = "ing-nginx-ingress-nginx-controller"
        port {
          name = "http"
        }
      }
    }
  }
}

resource "helm_release" "ing-nginx" {
  name       = "ing-nginx"

  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  namespace  = "kube-system"

  set {
    name  = "controller.service.type"
    value = "ClusterIP"
  }
  set {
    name  = "controller.publishService.enabled"
    value = "false"
  }
  set {
    name  = "controller.extraArgs.publish-status-address"
    value = kubernetes_ingress_v1.alb.status.0.load_balancer.0.ingress.0.hostname
  }
  set {
    name  = "controller.config.use-forwarded-headers"
    value = "true"
  }
  set {
    name  = "controller.ingressClassResource.default"
    value = "true"
  }
}

    It is a bit weird, as it introduces something like a circular dependency, but the ingress simply waits until nginx is finally up and all is well.

    This solution in not exactly the same as the --publish-ingress option as it will not be able to adapt to any changes of the ALB address. - Luckily I don't expect that address to change, so I'm fine with that solution.


  2. 0

    You can achieve this with two ingress controllers. The ALB ingress controller will hand the publicly exposed endpoint and route traffic to the nginx ingress controller as its backend. Then you configure your nginx ingress controller for managing ingress of application traffic.

    enter image description here

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search