Question 176935 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Can ARN be passed from ConfigMap to TargetGroupBinding in Kubernetes on AWS?

I need to do a Target Group Binding in AWS.
I am creating almost all resources using Terraform (EKS cluster, nodes, other AWS services).
On top of it I am having a Kubernetes cluster of course and code written in K8s yamls.

I am creating a service like f.e. nginx proxy which is meant to be dong its proxying job.

What I want to achieve is to bind the proxy service

kind: Service
metadata:
  name: nginx-proxy-service
  namespace: nginx-proxy
spec:
  selector:
    app: nginx-proxy
  ports:
    - protocol: TCP
      port: 443
      targetPort: 443

with a target group created in Terraform

resource "aws_lb_target_group" "nginx-proxy" {
  name        = "${var.environment}-proxy-tg"
  port        = 443
  protocol    = "HTTPS"
  vpc_id      = var.vpc_id
  target_type = "ip"
}

There is a CRD TargetGroupBinding that I can use for this purpose, but I need to pass the TargetGroup ARN to it

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: nginx-proxy-tgb
  namespace: nginx-proxy
spec:
  serviceRef:
    name: nginx-proxy-service
    port: 443
  targetGroupARN: $(TARGETGROUP_ARN)

Firstly I thought that I can use a ConfigMap, but that TargetGroupBinding doesn’t understand ConfigMaps.
Then I thought that I can use Kustomize, but I cannot figure out how (if it is possible) to pass a value from a ConfigMap that contains that value,

$ kubectl describe configmap proxy-cm
Name:         proxy-cm
Namespace:    nginx-proxy
Labels:       <none>
Annotations:  <none>

Data
====
targetgroup_arn:
----
arn:aws:elasticloadbalancing:eu-west-1:<somevaluehere>:targetgroup/beta-proxy-tg/<somevaluethere>

BinaryData
====

Events:  <none>

to a Kustomize, and then Kustomize can use it and replace proper fields in my TargetGroupBinding.

Do you have any ideas how can I get marry those two things?
Seems pretty common pattern, but as a newbie in Terraform and K8s I cannot figure it out.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I want to use Terraform to create infrastructure obcjets (EKS clusters, nodes, Load Balancers, Target groups and so on). I want to use kubernetes/Kustomize/Helm code to create K8s objects like deployments, services, statefulsets ad so on. I've almost done that using Helm chart approach

    Terraform object creation:

    resource "aws_lb_target_group" "nginx-proxy" {
  name        = "${var.environment}-proxy-tg"
  port        = 443
  protocol    = "HTTPS"
  vpc_id      = var.vpc_id
  target_type = "ip"
}

resource "kubernetes_config_map" "proxy_configmap" {
  metadata {
    name      = "proxy-cm"
    namespace = "nginx-proxy"
  }
  data = {
    targetgroup_arn  = "${aws_lb_target_group.nginx-proxy.arn}"
    lb-name = "${aws_lb.eks_alb.name}"
  }
  depends_on = [kubernetes_namespace.nginx-proxy-namespace]
}

    _helpers.tpl

    {{- define "awsLBTargetGroupArn" -}}
{{- (lookup "v1" "ConfigMap" "nginx-proxy" "proxy-cm").data.targetgroup_arn }}
{{- end -}}

    Helm templates:

    apiVersion: v1
kind: Service
metadata:
  name: nginx-proxy-service
  namespace: nginx-proxy
spec:
  selector:
    app: nginx-proxy
  ports:
    - protocol: TCP
      port: 443
      targetPort: 443

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: nginx-proxy-tgb
  namespace: nginx-proxy
spec:
  serviceRef:
    name: nginx-proxy-service
    port: 443
  targetGroupARN: {{ include "awsLBTargetGroupArn" . }}

    I can see that kubectl shows me that it is being binded

    ✗ kubectl describe TargetGroupBinding nginx-proxy-tgb
Name:         nginx-proxy-tgb
Namespace:    nginx-proxy
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: nginx-proxy
              meta.helm.sh/release-namespace: nginx-proxy
API Version:  elbv2.k8s.aws/v1beta1
Kind:         TargetGroupBinding
(...)



Ip Address Type:  ipv4
  Service Ref:
    Name:            nginx-proxy-service
    Port:            443
  Target Group ARN:  arn:aws:elasticloadbalancing:eu-west-1:<somenumbershere>:targetgroup/beta-proxy-tg/bba0b9519459370e
  Target Type:       ip
Status:
  Observed Generation:  1
Events:
  Type    Reason                  Age                From                Message
  ----    ------                  ----               ----                -------
  Normal  SuccessfullyReconciled  48m (x3 over 83m)  targetGroupBinding  Successfully reconciled

    But I can't see the nodes being attached in fact

    ✗ aws elbv2 describe-target-health  --target-group-arn arn:aws:elasticloadbalancing:eu-west-1:<somenumbershere>:targetgroup/beta-proxy-tg/bba0b9519459370e
----------------------
|DescribeTargetHealth|
+--------------------+

  2. 0

    If I understood the question, there is a need to apply a CRD to the cluster where one of the arguments should be populated by the value provided from another resource that gets created. For this purpose, the hashicorp/kubernetes provider can be used, namely its kubernetes_manifest resource. Based on the question it could look like the following:

    resource "aws_lb_target_group" "nginx-proxy" {
  name        = "${var.environment}-proxy-tg"
  port        = 443
  protocol    = "HTTPS"
  vpc_id      = var.vpc_id
  target_type = "ip"
}

resource "kubernetes_manifest" "target_group_binding_crd" {
  manifest = {
    apiVersion = "elbv2.k8s.aws/v1beta1"
    kind       = "TargetGroupBinding"

    metadata = {
      name      = "nginx-proxy-tgb"
      namespace = "nginx-proxy"
    }

    spec = {
      serviceRef = {
        name = "nginx-proxy-service"
        port = 443
      }
      targetGroupARN = aws_lb_target_group.nginx-proxy.arn
    }
  }
}

    Where the target group ARN would be passed to the kubernetes_manifest resource using implicit dependency, i.e., targetGroupARN = aws_lb_target_group.nginx-proxy.arn.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search