Getting 503 with ingress-nginx in k3s

I’ve been looking through a lot of posts of 503 errors when using ingress-nginx but haven’t come across a solution that works with my setup.

Running on a local Oracle Linux Server v8.5. Using k3s, and the following info for the node:

kubectl get node -o wide
NAME         STATUS   ROLES    AGE     VERSION        INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                  KERNEL-VERSION                    CONTAINER-RUNTIME
mptdevws00   Ready    master   6d21h   v1.19.2+k3s1   192.168.10.100   <none>        Oracle Linux Server 8.5   5.4.17-2102.201.3.el8uek.x86_64   containerd://1.4.0-k3s1 

Running the bare-metal ingress-nginx located at https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.4.0/deploy/static/provider/baremetal/deploy.yaml

I’m deploying a very basic hello-word deployment, service and ingress, defined in the following:

---
apiVersion: v1
kind: Service
metadata:
  name: hello-world-service
spec:
  type: NodePort
  ports:
    - port: 80
  selector:
    app:  hello-world

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-test
spec:
  selector:
    matchLabels:
      app: hello-world
  replicas: 1
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: hello-world-volume
          mountPath: /usr/share/nginx/html
      volumes:
      - name: hello-world-volume
        configMap:
          name: hello-world

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /hello
        pathType: Prefix
        backend:
          service:
            name: hello-world-service
            port:
              number: 80

Had to run mod_probe ip_tables to get that loaded, and I have to run kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission each time I launch the ingress-nginx deployment because the validation webhook continually spits out an Unknown Host error, so I just turned that piece off until I get this working.

Pods, services, ingress, and endpoints seem to launch fine:

kubectl get pods --all-namespaces -o wide
NAMESPACE       NAME                                       READY   STATUS             RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
kube-system     metrics-server-7b4f8b595-kp4j4             1/1     Running            0          6d21h   10.42.0.2     mptdevws00   <none>           <none>
kube-system     local-path-provisioner-7ff9579c6-s6f4z     1/1     Running            1          6d21h   10.42.0.4     mptdevws00   <none>           <none>
kube-system     coredns-66c464876b-4g5q6                   1/1     Running            0          6d21h   10.42.0.3     mptdevws00   <none>           <none>
ingress-nginx   svclb-ingress-nginx-controller-hc8gb       2/2     Running            0          58m     10.42.0.206   mptdevws00   <none>           <none>
ingress-nginx   ingress-nginx-admission-create-m9nfw       0/1     Completed          0          58m     10.42.0.207   mptdevws00   <none>           <none>
ingress-nginx   ingress-nginx-admission-patch-kdvdf        0/1     Completed          0          58m     10.42.0.208   mptdevws00   <none>           <none>
ingress-nginx   ingress-nginx-controller-5979dd67b-lrg5r   1/1     Running            0          58m     10.42.0.209   mptdevws00   <none>           <none>
default         hello-world-test-59cc4bc76f-b58k9          1/1     Running            0          57m     10.42.0.212   mptdevws00   <none>           <none>

kubectl get service --all-namespaces -o wide
NAMESPACE       NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE     SELECTOR
default         kubernetes                           ClusterIP      10.43.0.1       <none>           443/TCP                      6d21h   <none>
kube-system     kube-dns                             ClusterIP      10.43.0.10      <none>           53/UDP,53/TCP,9153/TCP       6d21h   k8s-app=kube-dns
kube-system     metrics-server                       ClusterIP      10.43.191.252   <none>           443/TCP                      6d21h   k8s-app=metrics-server
ingress-nginx   ingress-nginx-controller-admission   ClusterIP      10.43.141.95    <none>           443/TCP                      59m     app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx   ingress-nginx-controller             LoadBalancer   10.43.191.206   192.168.10.100   80:32700/TCP,443:31849/TCP   59m     app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
default         hello-world-service                  NodePort       10.43.56.86     <none>           80:31043/TCP                 58m     app=hello-world

kubectl get ingress --all-namespaces -o wide
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAMESPACE   NAME                  CLASS   HOSTS   ADDRESS          PORTS   AGE
default     hello-world-ingress   nginx   *       192.168.10.100   80      49m

kubectl get ep --all-namespaces -o wide
NAMESPACE       NAME                                 ENDPOINTS                                  AGE
default         kubernetes                           192.168.10.100:6443                        6d21h
kube-system     metrics-server                       10.42.0.2:443                              6d21h
kube-system     kube-dns                             10.42.0.3:53,10.42.0.3:9153,10.42.0.3:53   6d21h
ingress-nginx   ingress-nginx-controller             10.42.0.209:443,10.42.0.209:80             60m
ingress-nginx   ingress-nginx-controller-admission   10.42.0.209:8443                           60m
default         hello-world-service                  10.42.0.212:80                             59m
kube-system     rancher.io-local-path                <none>                                     6d21h

If I do a kubectl exec -it –namespace=ingress-nginx ingress-nginx-controller-5979dd67b-lrg5r — curl http://hello-world-service.default.svc/ it returns the basic Hello World html fine, so I know the service and deployment are at least working correctly.

If I do curl http://localhost/ (or using the host ip, or hostname) I get a 404, which is what I would expect based on the nginx.conf response for /. So that tells me the ingress point is at least up and running. http://localhost/healthz returns a 200, which again I would expect based on nginx.conf

However, doing curl http://localhost/hello always returns a 503.

In the controller logs I just get:

I1011 20:34:13.274379       8 controller.go:168] "Configuration changes detected, backend reload required"
I1011 20:34:13.406122       8 controller.go:185] "Backend successfully reloaded"
I1011 20:34:13.407450       8 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5979dd67b-lrg5r", UID:"49499847-4f0f-4196-ba71-81980445b46c", APIVersion:"v1", ResourceVersion:"444384", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W1011 20:34:21.685551       8 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: the server could not find the requested resource
E1011 20:34:21.686258       8 reflector.go:140] k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: the server could not find the requested resource
I1011 20:34:30.136621       8 store.go:430] "Found valid IngressClass" ingress="default/hello-world-ingress" ingressclass="nginx"
W1011 20:34:30.137527       8 endpointslices.go:81] Error obtaining Endpoints for Service "default/hello-world-service": no object matching key "default/hello-world-service" in local store
W1011 20:34:30.137569       8 controller.go:1112] Service "default/hello-world-service" does not have any active Endpoint.
I1011 20:34:30.137664       8 controller.go:168] "Configuration changes detected, backend reload required"
I1011 20:34:30.142794       8 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"hello-world-ingress", UID:"3d297335-e2eb-46cf-a6a5-56bf68725c8f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"447873", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1011 20:34:30.250894       8 controller.go:185] "Backend successfully reloaded"
I1011 20:34:30.251447       8 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5979dd67b-lrg5r", UID:"49499847-4f0f-4196-ba71-81980445b46c", APIVersion:"v1", ResourceVersion:"444384", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1011 20:34:58.695644       8 status.go:299] "updating Ingress status" namespace="default" ingress="hello-world-ingress" currentValue=[] newValue=[{IP:192.168.10.100 Hostname: Ports:[]}]
W1011 20:34:58.702687       8 endpointslices.go:81] Error obtaining Endpoints for Service "default/hello-world-service": no object matching key "default/hello-world-service" in local store
W1011 20:34:58.702735       8 controller.go:1112] Service "default/hello-world-service" does not have any active Endpoint.
I1011 20:34:58.703331       8 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"hello-world-ingress", UID:"3d297335-e2eb-46cf-a6a5-56bf68725c8f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"447926", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W1011 20:34:59.764277       8 reflector.go:424] k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: the server could not find the requested resource
E1011 20:34:59.764372       8 reflector.go:140] k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: the server could not find the requested resource
10.42.0.206 - - [11/Oct/2022:20:35:19 +0000] "GET /hello HTTP/1.1" 503 190 "-" "curl/7.61.1" 78 0.000 [default-hello-world-service-80] [] - - - - 3cc8c307cea5d96642b37e10a01707f1

Doesn’t matter if I add a rewrite-target of /, /%2, etc. I’ve tried defining a type: NodePort for my service, or leaving that off. Doesn’t change anything. Seems no matter what the various other problems people had, I can’t seem to get this to work.

I’ve also tried execing into various containers to hit ips of the service, service name, all that, and that seems fine, it’s just the path between the ingress point and the service is failing. Turning on debug in my ingress-nginx deployment didn’t provide any details that pointed to the issue (at least that I could fathom).

Would greatly appreciate any help that might direct me to a solution, or least help track down where the breakage is occurring. Logs in the actual hello-world pod show successful requests when I run curl from within a pod, but show nothing when I get the 503’s, so the request isn’t even hitting the service.