Ingress rewrite rule in aks agic gives 502 – Nginx

I’m trying to create HTTPS ingress for my node.js authentication (auth) REST service in AKS, but I’m getting a 502 Bad Gateway response.

Here’s my deployment and service definitions:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth
  namespace: auth
  labels:
    app: auth
spec:
  selector:
    matchLabels:
      app: auth
  replicas: 1
  template:
    metadata:
      labels:
        app: auth
    spec:
      imagePullSecrets:
        - name: docker-hub-creds
      containers:
      - name: auth
        image:  ***image***
        ports:
        - containerPort:  80
          name: auth
---
apiVersion: v1
kind: Service
metadata:
  name: auth
  namespace: auth
spec:
  selector:
    app: auth
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

I think that’s all pretty basic and it seems to work ok. I can see the service running and if I expose a node-port then I can access it with no problems. The service responds to well-formed POST requests on the /auth path with a JWT.

I have configured an Azure Application Gateway following Microsoft’s instructions, and following the troubleshooting guide leads me to believe that the installation has worked ok. I have also checked through the web-ui and there appear to be no errors. Finally, I worked through the support options and the automated analysis of my cluster found no major configuration issues.

Next, I tried to create an HTTPS ingress route for my service, and this is where it goes wrong. This is made more complicated by the dynamic generation of certificates for TLS.

The ingress definition looks like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: auth-in
  namespace: auth
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt-staging
    cert-manager.io/acme-challenge-type: http01
    ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  tls:
  - hosts:
      - ***hostname***
    secretName: ***secret***
  rules:
  - host: ***same hostname***
    http:
      paths:
      - backend:
          serviceName: auth
          servicePort: 80
        path: /api/(auth/.*)

I have two rewrite-targets in there because I can’t determine which one this ingress controller uses. All the example from the web use the nginx. prefix so I added it in desperation, despite thinking that it’s probably not necessary.

Accessing the service through: ***hostname***/api/auth results in a Bad Gateway error.

I have checked through the portal and I can see the route is registered, listeners and rules are there, and my service is listed in the backend pools, but there is nothing in the ‘rewrite’ tabs. I expected to see something in the rewrite tabs.

I’ve tooled my service to log all access, and the logs show this, repeatedly:

{"level":30,"time":1611739355140,"pid":17,"hostname":"auth-6c7757bb89-d72td","msg":"Req-URL: /api/(auth/.*)"}

Describing the ingress gives me this:

Name:             auth-in
Namespace:        auth
Address:          **redacted***
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  ***redacted cert name** terminates **hostname***
Rules:
  Host                       Path  Backends
  ----                       ----  --------
  ***hostname***
                             /api/(auth/.*)   auth:80   10.0.0.69:80)
Annotations:                 cert-manager.io/acme-challenge-type: http01
                             cert-manager.io/cluster-issuer: letsencrypt-staging
                             ingress.kubernetes.io/rewrite-target: /$1
                             kubernetes.io/ingress.class: azure/application-gateway
                             nginx.ingress.kubernetes.io/rewrite-target: /$1
Events:
  Type    Reason             Age   From          Message
  ----    ------             ----  ----          -------
  Normal  CreateCertificate  43m   cert-manager  Successfully created Certificate "***cert-name***"

Two things to note. 1st that the logs show that the URL isn’t being rewritten — it’s being passed exactly as the path shows, including the regex part. 2nd, that the Default Backend entry in the ingress description shows an error. I’m not sure that the 2nd one matters, but the 1st is clearly wrong.

I am keen to discover how to diagnose the problem and then fix it.