skip to Main Content

I am running a 2-node K8s cluster on OVH Bare Metal Servers. I’ve set up MetalLB and Nginx-Ingress.The 2 servers both have public IPs and are not in the same network segment. I’ve used one of the IPs as the entrypoint for the LB. The deployments I created 3 nginx containers & services to test the forwarding.
When I use host based routing, the endpoints are reachable via the internet, but when I use path based forwarding, only the / path is reachable. For the rest, I get the default backend.
My host based Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource-2
spec:
  ingressClassName: nginx
  rules:
  - host: nginx.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-main
            port:
              number: 80
  - host: blue.nginx.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-blue
            port:
              number: 80
  - host: green.nginx.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-green
            port:
              number: 80

The path based Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource-3
spec:
  ingressClassName: nginx
  rules:
  - host: nginx.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80
      - path: /blue
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-blue
            port:
              number: 80
      - path: /green
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-green
            port:
              number: 80

The endpoints are all reachable in both cases

# kubectl describe ing ingress-resource-2
Name:             ingress-resource-2
Labels:           <none>
Namespace:        default
Address:          
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                     Path  Backends
  ----                     ----  --------
  nginx.example.com        
                           /   nginx:80 (192.168.107.4:80)
  blue.nginx.example.com   
                           /   nginx-deploy-blue:80 (192.168.164.212:80)
  green.nginx.example.com  
                           /   nginx-deploy-green:80 (192.168.164.213:80)
Annotations:               <none>
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  13m   nginx-ingress-controller  Configuration for default/ingress-resource-2 was added or updated

# kubectl describe ing ingress-resource-3
Name:             ingress-resource-3
Labels:           <none>
Namespace:        default
Address:          
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host               Path  Backends
  ----               ----  --------
  nginx.example.com  
                     /        nginx:80 (192.168.107.4:80)
                     /blue    nginx-deploy-blue:80 (192.168.164.212:80)
                     /green   nginx-deploy-green:80 (192.168.164.213:80)
Annotations:         <none>
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  109s  nginx-ingress-controller  Configuration for default/ingress-resource-3 was added or updated

Getting the Nginx-Ingress logs:

# kubectl -n nginx-ingress logs pod/nginx-ingress-6947fb84d4-m9gkk

W0803 17:00:48.516628       1 flags.go:273] Ignoring unhandled arguments: []
I0803 17:00:48.516688       1 flags.go:190] Starting NGINX Ingress Controller Version=2.3.0 PlusFlag=false
I0803 17:00:48.516692       1 flags.go:191] Commit=979db22d8065b22fedb410c9b9c5875cf0a6dc66 Date=2022-07-12T08:51:24Z DirtyState=false Arch=linux/amd64 Go=go1.18.3
I0803 17:00:48.527699       1 main.go:210] Kubernetes version: 1.24.3
I0803 17:00:48.531079       1 main.go:326] Using nginx version: nginx/1.23.0
2022/08/03 17:00:48 [notice] 26#26: using the "epoll" event method
2022/08/03 17:00:48 [notice] 26#26: nginx/1.23.0
2022/08/03 17:00:48 [notice] 26#26: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2022/08/03 17:00:48 [notice] 26#26: OS: Linux 5.15.0-41-generic
2022/08/03 17:00:48 [notice] 26#26: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/08/03 17:00:48 [notice] 26#26: start worker processes
2022/08/03 17:00:48 [notice] 26#26: start worker process 27
2022/08/03 17:00:48 [notice] 26#26: start worker process 28
2022/08/03 17:00:48 [notice] 26#26: start worker process 29
2022/08/03 17:00:48 [notice] 26#26: start worker process 30
2022/08/03 17:00:48 [notice] 26#26: start worker process 31
2022/08/03 17:00:48 [notice] 26#26: start worker process 32
2022/08/03 17:00:48 [notice] 26#26: start worker process 33
2022/08/03 17:00:48 [notice] 26#26: start worker process 34
I0803 17:00:48.543403       1 listener.go:54] Starting Prometheus listener on: :9113/metrics
2022/08/03 17:00:48 [notice] 26#26: start worker process 35
2022/08/03 17:00:48 [notice] 26#26: start worker process 37
I0803 17:00:48.543712       1 leaderelection.go:248] attempting to acquire leader lease nginx-ingress/nginx-ingress-leader-election...
2022/08/03 17:00:48 [notice] 26#26: start worker process 38
...
2022/08/03 17:00:48 [notice] 26#26: start worker process 86
I0803 17:00:48.645253       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"delivery-ingress", UID:"23f93b2d-c3c8-48eb-a2a1-e2ce0453677f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1527358", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/delivery-ingress was added or updated 
I0803 17:00:48.645512       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated 
I0803 17:00:48.646550       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated 
I0803 17:00:48.646629       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"delivery-ingress", UID:"23f93b2d-c3c8-48eb-a2a1-e2ce0453677f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1527358", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/delivery-ingress was added or updated 
I0803 17:00:48.646810       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated 
I0803 17:00:48.646969       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated 
I0803 17:00:48.647259       1 event.go:285] Event(v1.ObjectReference{Kind:"Secret", Namespace:"nginx-ingress", Name:"default-server-secret", UID:"d8271053-2785-408f-b87b-88b9bb9fc488", APIVersion:"v1", ResourceVersion:"1612716", FieldPath:""}): type: 'Normal' reason: 'Updated' the special Secret nginx-ingress/default-server-secret was updated
2022/08/03 17:00:48 [notice] 26#26: signal 1 (SIGHUP) received from 88, reconfiguring
2022/08/03 17:00:48 [notice] 26#26: reconfiguring
2022/08/03 17:00:48 [notice] 26#26: using the "epoll" event method
2022/08/03 17:00:48 [notice] 26#26: start worker processes
2022/08/03 17:00:48 [notice] 26#26: start worker process 89
2022/08/03 17:00:48 [notice] 26#26: start worker process 90
...
2022/08/03 17:00:48 [notice] 26#26: start worker process 136
2022/08/03 17:00:48 [notice] 27#27: gracefully shutting down
2022/08/03 17:00:48 [notice] 27#27: exiting
2022/08/03 17:00:48 [notice] 35#35: gracefully shutting down
2022/08/03 17:00:48 [notice] 31#31: exiting
2022/08/03 17:00:48 [notice] 38#38: gracefully shutting down
2022/08/03 17:00:48 [notice] 32#32: exiting
2022/08/03 17:00:48 [notice] 30#30: exiting
2022/08/03 17:00:48 [notice] 40#40: gracefully shutting down
2022/08/03 17:00:48 [notice] 35#35: exiting
2022/08/03 17:00:48 [notice] 45#45: gracefully shutting down
2022/08/03 17:00:48 [notice] 40#40: exiting
2022/08/03 17:00:48 [notice] 48#48: gracefully shutting down
2022/08/03 17:00:48 [notice] 47#47: exiting
2022/08/03 17:00:48 [notice] 57#57: gracefully shutting down
2022/08/03 17:00:48 [notice] 52#52: exiting
2022/08/03 17:00:48 [notice] 55#55: gracefully shutting down
2022/08/03 17:00:48 [notice] 55#55: exiting
2022/08/03 17:00:48 [notice] 51#51: gracefully shutting down
2022/08/03 17:00:48 [notice] 51#51: exiting
2022/08/03 17:00:48 [notice] 31#31: exit
2022/08/03 17:00:48 [notice] 34#34: gracefully shutting down
2022/08/03 17:00:48 [notice] 34#34: exiting
2022/08/03 17:00:48 [notice] 41#41: exiting
2022/08/03 17:00:48 [notice] 49#49: gracefully shutting down
....
2022/08/03 17:00:48 [notice] 49#49: exiting
2022/08/03 17:00:48 [notice] 57#57: exit
.....
2022/08/03 17:00:48 [notice] 43#43: exit
2022/08/03 17:00:48 [notice] 58#58: gracefully shutting down
2022/08/03 17:00:48 [notice] 38#38: exiting
2022/08/03 17:00:48 [notice] 53#53: gracefully shutting down
2022/08/03 17:00:48 [notice] 48#48: exiting
2022/08/03 17:00:48 [notice] 59#59: gracefully shutting down
2022/08/03 17:00:48 [notice] 58#58: exiting
2022/08/03 17:00:48 [notice] 62#62: gracefully shutting down
2022/08/03 17:00:48 [notice] 60#60: gracefully shutting down
2022/08/03 17:00:48 [notice] 53#53: exiting
2022/08/03 17:00:48 [notice] 61#61: gracefully shutting down
2022/08/03 17:00:48 [notice] 63#63: gracefully shutting down
2022/08/03 17:00:48 [notice] 64#64: gracefully shutting down
2022/08/03 17:00:48 [notice] 59#59: exiting
2022/08/03 17:00:48 [notice] 65#65: gracefully shutting down
2022/08/03 17:00:48 [notice] 62#62: exiting
2022/08/03 17:00:48 [notice] 60#60: exiting
2022/08/03 17:00:48 [notice] 66#66: gracefully shutting down
2022/08/03 17:00:48 [notice] 67#67: gracefully shutting down
2022/08/03 17:00:48 [notice] 63#63: exiting
2022/08/03 17:00:48 [notice] 68#68: gracefully shutting down
2022/08/03 17:00:48 [notice] 64#64: exiting
2022/08/03 17:00:48 [notice] 61#61: exiting
2022/08/03 17:00:48 [notice] 69#69: gracefully shutting down
2022/08/03 17:00:48 [notice] 65#65: exiting
2022/08/03 17:00:48 [notice] 66#66: exiting
2022/08/03 17:00:48 [notice] 71#71: gracefully shutting down
2022/08/03 17:00:48 [notice] 70#70: gracefully shutting down
2022/08/03 17:00:48 [notice] 67#67: exiting
...
2022/08/03 17:00:48 [notice] 65#65: exit
2022/08/03 17:00:48 [notice] 73#73: gracefully shutting down
...
2022/08/03 17:00:48 [notice] 74#74: exiting
2022/08/03 17:00:48 [notice] 83#83: gracefully shutting down
2022/08/03 17:00:48 [notice] 72#72: exiting
2022/08/03 17:00:48 [notice] 77#77: gracefully shutting down
2022/08/03 17:00:48 [notice] 77#77: exiting
2022/08/03 17:00:48 [notice] 77#77: exit
I0803 17:00:48.780547       1 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"nginx-ingress", Name:"nginx-config", UID:"961b1b89-3765-4eb8-9f5f-cfd8212012a8", APIVersion:"v1", ResourceVersion:"1612730", FieldPath:""}): type: 'Normal' reason: 'Updated' Configuration from nginx-ingress/nginx-config was updated 
I0803 17:00:48.780573       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"delivery-ingress", UID:"23f93b2d-c3c8-48eb-a2a1-e2ce0453677f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1527358", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/delivery-ingress was added or updated 
I0803 17:00:48.780585       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated 
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 72
2022/08/03 17:00:48 [notice] 26#26: worker process 72 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 30
2022/08/03 17:00:48 [notice] 26#26: worker process 30 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 35 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 77 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 73
2022/08/03 17:00:48 [notice] 26#26: worker process 73 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 37
2022/08/03 17:00:48 [notice] 26#26: worker process 29 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 32 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 37 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 38 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 41 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 47 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 49 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 63 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 64 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 75 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 47
2022/08/03 17:00:48 [notice] 26#26: worker process 34 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 43 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 48 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 53 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 54 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 59 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 61 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 66 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 55
2022/08/03 17:00:48 [notice] 26#26: worker process 50 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 55 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 83
2022/08/03 17:00:48 [notice] 26#26: worker process 28 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 31 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 42 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 51 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 52 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 56 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 62 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 68 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 71 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 83 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 33
2022/08/03 17:00:48 [notice] 26#26: worker process 33 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 58
2022/08/03 17:00:48 [notice] 26#26: worker process 58 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 57
2022/08/03 17:00:48 [notice] 26#26: worker process 27 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 57 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 40
2022/08/03 17:00:48 [notice] 26#26: worker process 40 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 45 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 60 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 65 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 67 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 69 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 70 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 74 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 86 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received

I’m not sure what the issue is, and I can’t figure out why it’s working when I use different hosts, and not working when I try to use different paths.

I thought it could be resource limits, but I only have the requests, no limits. There is already a default IngressClass

I installed the ingress controller via manifests following the steps here

Update: To add the deployments running in the cluster.

# nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-main
  template:
    metadata:
      labels:
        run: nginx-main
    spec:
      containers:
      - image: nginx
        name: nginx
# nginx-deploy-green.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx-deploy-green
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-green
  template:
    metadata:
      labels:
        run: nginx-green
    spec:
      volumes:
      - name: webdata
        emptyDir: {}
      initContainers:
      - name: web-content
        image: busybox
        volumeMounts:
        - name: webdata
          mountPath: "/webdata"
        command: ["/bin/sh", "-c", 'echo "<h1>I am <font color=green>GREEN</font></h1>" > /webdata/index.html']
      containers:
      - image: nginx
        name: nginx
        volumeMounts:
        - name: webdata
          mountPath: "/usr/share/nginx/html"
# nginx-deploy-blue.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx-deploy-blue
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-blue
  template:
    metadata:
      labels:
        run: nginx-blue
    spec:
      volumes:
      - name: webdata
        emptyDir: {}
      initContainers:
      - name: web-content
        image: busybox
        volumeMounts:
        - name: webdata
          mountPath: "/webdata"
        command: ["/bin/sh", "-c", 'echo "<h1>I am <font color=blue>BLUE</font></h1>" > /webdata/index.html']
      containers:
      - image: nginx
        name: nginx
        volumeMounts:
        - name: webdata
          mountPath: "/usr/share/nginx/html"

3

Answers


  1. Chosen as BEST ANSWER

    I'm not sure if the issue was in how I had deployed the NGINX Ingress Controller, but I tried using the Ingress-nginx controller, and the ingress objects work perfectly well.

    I will still try again with the NGINX Ingress Controller just to be sure, but for now the routing is behaving as expected with the other ingress controller.

    Thanks for your help guys. I will update if I get it to work with the original Ingress Controller.


  2. I believe the reason is that the nginx service you have deployed for "blue" and "green" deployments are not serving anything on the /blue and /green endpoints. You need to create these endpoints inside nginx so that they become valid. Your ingress objects appear fine.

    That’s also why you see / working in all cases since by default nginx is hosting a web page on the path / but not on the paths /blue and /green. Simply creating the respective endpoints in your services for nginx should fix the problem.

    Login or Signup to reply.
  3. Based on the comments from zer0 answered, try:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: ingress-resource-3
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: /  # <-- add
    spec:
      ingressClassName: nginx
      ...
    

    Page with different font color should response when your browse to http://nginx.example.com/blue or green

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search