Nginx – With Terraform, how do I integrate a basic-sku load balancer and basic-sku public ip address with an azurerm_kubernetes_cluster resource?

The below allows for a basic sku LB and IP to be used within an AKS cluster whose load_balancer_sku = basic.

The answer is that the above pattern is obsolete. We used it to establish a consistent public IP address that would never change (for DNS and other purposes), and we attached it to a custom load balancer.

This allowed reconfiguring of the load balancer in a way that would not destroy the entire cluster. The load balancer and custom public ip could exist in its own resource group, and giving the SystemAssigned managed identity access to it via "Contributor" allowed it all to work.

The documentation states that Node Resource Group resources, which are auto-managed by Microsoft, cannot be altered. And now, per Azurerm provider 3.0, this includes the Load Balancer. In other words, the LB is now automatically created and included in the NRG, so it is no longer possible to create a load balancer resource that works with an aks cluster.

However, you can still create a custom Public IP address, which, with the right configuration, can be used with the DNS zones, etc. The pattern now is to create a custom public ip address and assign it to the auto-managed NRG. Then, you are forced to use Microsoft's annotations to implement it, like this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-resource-group: <node resource group name>
    service.beta.kubernetes.io/azure-pip-name: myAKSPublicIP
  name: azure-load-balancer
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: azure-load-balancer

From a terraform perspective, here is what it all looks like put together:

# 1.create the resource group

# az group create --name myNetworkResourceGroup --location eastus
resource "azurerm_resource_group" "rg" {
  name     = "${var.name}-rg"
  location = var.location
}

# 2. create the BASIC cluster
# az aks create --name myAKSCluster --resource-group myNetworkResourceGroup --generate-ssh-keys --tier free --node-count 1
resource "azurerm_kubernetes_cluster" "k8s" {
  location            = azurerm_resource_group.rg.location
  name                = var.name
  resource_group_name = azurerm_resource_group.rg.name
  node_resource_group = local.node_resource_group_name
  dns_prefix          = local.dns_prefix

  identity {
    type = "SystemAssigned"
  }

  default_node_pool {
    name       = "agentpool"
    vm_size    = "Standard_B2s"
    node_count = 1
  }

  network_profile {
    network_plugin    = "kubenet"
    load_balancer_sku = "basic" # var.sku naming error # ref: https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-overview#pricing-and-sla
  }
}

# create the custom public ip address and insert it into the NRG
resource "azurerm_public_ip" "k8s_lb_ip" {
  name                = "PublicIPForLB"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_kubernetes_cluster.k8s.node_resource_group # NOTE IMPORTANT THIS MUST BE THE NODE RESOURCE GROUP NAME!
  allocation_method   = "Static"
  sku                 = "Basic"
}

# assign the kublet id role to the node resource group id itself (which seems counter intuitive)
# ref: https://learn.microsoft.com/en-us/azure/aks/static-ip?source=recommendations#create-a-service-using-the-static-ip-address
resource "azurerm_role_assignment" "lb_ip_aks_integration" {
  description          = "Allows a basic-sku load balancer and basic-sku Public IP integration with an AKS cluster whose load_balancer_sku = basic."
  principal_id         = azurerm_kubernetes_cluster.k8s.kubelet_identity[0].object_id
  role_definition_name = "Network Contributor"
  scope                            = azurerm_kubernetes_cluster.k8s.node_resource_group_id
  skip_service_principal_aad_check = true
}

unlike in versions 2.99 and earlier where we explicitely involve the load balancer details, we omit it all together. Instead, because the LB is auto created, we have to use annotations, so in Terraform, like this:

resource "kubernetes_manifest" "service" {
  depends_on = [module.aks_cluster]
  manifest = yamldecode(<<YAML
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-resource-group: autolbtest-nrg
    service.beta.kubernetes.io/azure-pip-name: PublicIPForLB
  name: azure-load-balancer
  namespace: default
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: azure-load-balancer
YAML
  )
}

TODO: use interpolation for the public ip name and the name of the resource group into which the public ip is created