How to expose a service to outside Kubernetes cluster via ingress? – Nginx

I’m struggling to expose a service in an AWS cluster to outside and access it via a browser. Since my previous question haven’t drawn any answers, I decided to simplify the issue in several aspects.

First, I’ve created a deployment which should work without any configuration. Based on this article, I did

1. kubectl create namespace tests

2. created file probe-service.yaml based on paulbouwer/hello-kubernetes:1.8 and deployed it kubectl create -f probe-service.yaml -n tests:

   apiVersion: v1
   kind: Service
   metadata:
     name: hello-kubernetes-first
   spec:
     type: ClusterIP
     ports:
     - port: 80
       targetPort: 8080
     selector:
       app: hello-kubernetes-first
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: hello-kubernetes-first
   spec:
     replicas: 3
     selector:
       matchLabels:
         app: hello-kubernetes-first
     template:
       metadata:
         labels:
           app: hello-kubernetes-first
       spec:
         containers:
         - name: hello-kubernetes
           image: paulbouwer/hello-kubernetes:1.8
           ports:
           - containerPort: 8080
           env:
           - name: MESSAGE
             value: Hello from the first deployment!
   

3. created ingress.yaml and applied it (kubectl apply -f .probesingress.yaml -n tests)

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hello-kubernetes-ingress
   spec:
     rules:
     - host: test.projectname.org
       http:
         paths:
         - pathType: Prefix
           path: "/test"
           backend:
             service:
               name: hello-kubernetes-first
               port:
                 number: 80
     - host: test2.projectname.org
       http:
         paths:
         - pathType: Prefix
           path: "/test2"
           backend:
             service:
               name: hello-kubernetes-first
               port:
                 number: 80
     ingressClassName: nginx
   

Second, I can see that DNS actually point to the cluster and ingress rules are applied:

• if I open http://test.projectname.org/test or any irrelevant path (http://test.projectname.org/test3), I’m shown NET::ERR_CERT_AUTHORITY_INVALID, but
• if I use "open anyway" in browser, irrelevant paths give ERR_TOO_MANY_REDIRECTS while http://test.projectname.org/test gives Cannot GET /test

Now, TLS issues aside (those deserve a separate question), why can I get Cannot GET /test? It looks like ingress controller (ingress-nginx) got the rules (otherwise it wouldn’t descriminate paths; that’s why I don’t show DNS settings, although they are described in the previous question) but instead of showing the simple hello-kubernetes page at /test it returns this simple 404 message. Why is that? What could possibly go wrong? How to debug this?

Some debug info:

• kubectl version --short tells Kubernetes Client Version is v1.21.5 and Server Version is v1.20.7-eks-d88609

• kubectl get ingress -n tests shows that hello-kubernetes-ingress exists indeed, with nginx class, 2 expected hosts, address equal to that shown for load balancer in AWS console

• kubectl get all -n tests shows

  NAME                                          READY   STATUS    RESTARTS   AGE
  pod/hello-kubernetes-first-6f77d8ff99-gjw5d   1/1     Running   0          5h4m
  pod/hello-kubernetes-first-6f77d8ff99-ptwsn   1/1     Running   0          5h4m
  pod/hello-kubernetes-first-6f77d8ff99-x8w87   1/1     Running   0          5h4m
  
  NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
  service/hello-kubernetes-first   ClusterIP   10.100.18.189   <none>        80/TCP    5h4m
  
  NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
  deployment.apps/hello-kubernetes-first   3/3     3            3           5h4m
  
  NAME                                                DESIRED   CURRENT   READY   AGE
  replicaset.apps/hello-kubernetes-first-6f77d8ff99   3         3         3       5h4m
  

• ingress-nginx was installed before me via the following chart:

  apiVersion: v2
  name: nginx
  description: A Helm chart for Kubernetes
  type: application
  version: 4.0.6
  appVersion: "1.0.4"
  dependencies:
  - name: ingress-nginx
    version: 4.0.6
    repository: https://kubernetes.github.io/ingress-nginx
  

  and the values overwrites applied with the chart differ from the original ones mostly (well, those got updated since the installation) in extraArgs: default-ssl-certificate: "nginx-ingress/dragon-family-com" is uncommneted

PS To answer Andrew, I indeed tried to setup HTTPS but it seemingly didn’t help, so I haven’t included what I tried into the initial question. Yet, here’s what I did:

1. installed cert-manager, currently without a custom chart: kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml

2. based on cert-manager’s tutorial and SO question created a ClusterIssuer with the following config:

   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: letsencrypt-backoffice
   
   spec:
     acme:
       server: https://acme-staging-v02.api.letsencrypt.org/directory
       # use https://acme-v02.api.letsencrypt.org/directory after everything is fixed and works
       privateKeySecretRef: # this secret will be created in the namespace of cert-manager
         name: letsencrypt-backoffice-private-key
       # email: <will be used for urgent alerts about expiration etc>
   
       solvers:
       # TODO: add for each domain/second-level domain/*.projectname.org
       - selector:
           dnsZones:
             - test.projectname.org
             - test2.projectname.org
         # haven't made it to work yet, so switched to the simpler to configure http01 challenge
         # dns01:
         #   route53:
         #     region: ... # that of load balancer (but we also have ...)
         #     accessKeyID: <of IAM user with access to Route53>
         #     secretAccessKeySecretRef: # created that
         #       name: route53-credentials-secret
         #       key: secret-access-key
         #     role: arn:aws:iam::645730347045:role/cert-manager
         http01:
           ingress:
             class: nginx
   

   and applied it via kubectl apply -f issuer.yaml

3. created 2 certificates in the same file and applied it again:

   ---
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: letsencrypt-certificate
   spec:
     secretName: tls-secret
     issuerRef:
       kind: ClusterIssuer
       name: letsencrypt-backoffice
     commonName: test.projectname.org
     dnsNames:
     - test.projectname.org
   ---
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: letsencrypt-certificate-2
   spec:
     secretName: tls-secret-2
     issuerRef:
       kind: ClusterIssuer
       name: letsencrypt-backoffice
     commonName: test2.projectname.org
     dnsNames:
     - test2.projectname.org
   

4. made sure that the certificates are issued correctly (skipping the pain part, the result is: kubectl get certificates shows that both certificates have READY = true and both tls secrets are created)

5. figured that my ingress is in another namespace and secrets for tls in ingress spec can only be referred in the same namespace (haven’t tried the wildcard certificate and --default-ssl-certificate option yet), so for each one copied them to tests namespace:

   1. opened existing secret, like kubectl edit secret tls-secret-2, copied data and annotations
   2. created an empty (Opaque) secret in tests: kubectl create secret generic tls-secret-2-copy -n tests
   3. opened it (kubectl edit secret tls-secret-2-copy -n tests) and inserted data an annotations

6. in ingress spec, added the tls bit:

   tls:
   - hosts:
     - test.projectname.org
     secretName: tls-secret-copy
   - hosts:
     - test2.projectname.org
     secretName: tls-secret-2-copy
   

7. I hoped that this will help, but actually it made no difference (I get ERR_TOO_MANY_REDIRECTS for irrelevant paths, redirect from http to https, NET::ERR_CERT_AUTHORITY_INVALID at https and Cannot GET /test if I insist on getting to the page)