skip to Main Content

As stated in the title, I currently have a configuration with 2 ingress-nginx v1.0.0 on gke v1.20.10.

When I deploy one alone the configuration is working and I have no issue, but when I deploy the second one the validatingwebhook and then try to deploy an ingress the 2 validatingwebhook try to evaluate the newly created ingress.

This result in this error:

**Error from server (InternalError): error when creating "ingress-example.yaml": Internal error occurred: failed calling webhook "validate.nginx-public.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission-public.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": x509: certificate is valid for ingress-nginx-controller-admission-private, ingress-nginx-controller-admission-private.ingress-nginx.svc, not ingress-nginx-controller-admission-public.ingress-nginx.svc**

I checked and everything seems to be correctly separated, my validatingwebhook is deployed like that, the {{ ingress_type }} is a placeholder for -public or -private:

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx{{ ingress_type }}
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.0
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission{{ ingress_type }}
webhooks:
  - name: validate.nginx{{ ingress_type }}.ingress.kubernetes.io
    matchPolicy: Equivalent
    objectSelector:
      matchLabels:
        ingress-nginx : nginx{{ ingress_type }}
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission{{ ingress_type }}
        path: /networking/v1/ingresses
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx{{ ingress_type }}
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.0
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission{{ ingress_type }}
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx{{ ingress_type }}

I can’t seem to find a solution, there is an old github issue on that with no answer, maybe I’m doing something wrong but I just can’t see it.

As asked in comment, here is the ingress-example I’m trying to deploy, this works perfectly fine with only one ingress, not with two:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    kubernetes.io/ingress.class: nginx-private
#    external-dns.alpha.kubernetes.io/target: "IP"
  labels:
    ingress-nginx : nginx-public
spec:
  rules:
    - host: hello.MYDOMAINHERE
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 8080

2

Answers


  1. Chosen as BEST ANSWER

    So for those that may encounter this error.

    I tried different things before finding what was wrong. You have to rename all the labels but the version of the ingress-nginx, I did not think that it would break for so little, but it does. In the end I'm using something like this:

    ---
    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingWebhookConfiguration
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx{{ ingress_type }}
        app.kubernetes.io/instance: ingress-nginx{{ ingress_type }}
        app.kubernetes.io/version: 1.0.0
        app.kubernetes.io/component: admission-webhook{{ ingress_type }}
      name: ingress-nginx-admission{{ ingress_type }}
    webhooks:
      - name: validate.nginx{{ ingress_type }}.ingress.kubernetes.io
        matchPolicy: Equivalent
        objectSelector:
          matchLabels:
            ingress-nginx : nginx{{ ingress_type }}
        rules:
          - apiGroups:
              - networking.k8s.io
            apiVersions:
              - v1
            operations:
              - CREATE
              - UPDATE
            resources:
              - ingresses
        failurePolicy: Fail
        sideEffects: None
        admissionReviewVersions:
          - v1
        clientConfig:
          service:
            namespace: ingress-nginx
            name: ingress-nginx-controller-admission{{ ingress_type }}
            path: /networking/v1/ingresses
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx{{ ingress_type }}
        app.kubernetes.io/instance: ingress-nginx{{ ingress_type }}
        app.kubernetes.io/version: 1.0.0
        app.kubernetes.io/component: controller{{ ingress_type }}
      name: ingress-nginx-controller-admission{{ ingress_type }}
    spec:
      type: ClusterIP
      ports:
        - name: https-webhook
          port: 443
          targetPort: webhook
          appProtocol: https
      selector:
        app.kubernetes.io/name: ingress-nginx{{ ingress_type }}
    

    I think in this case it's really important to do the same on all the resources.


  2. did this solution work for having Webhook validating base on the ingressClass that is specified in the ingrass.yaml object? I guess not.
    In my case, lets say I have :
    NamespaceA , IngressControllerA with ingressClassA and ValidatingWebHookA
    and
    in a different namespace
    NamespaceB , IngressControllerB with ingressClassB and ValidatingWebHookB

    Now, if I create ingress with ingressClassA. ValidatingWebHookB seems to validating it and admitting it to IngressControllerA . Which is fine but if I remove ingressControllerB , then It wont work anymore. Complaining serviceB for the validatingWebhookB is not available.
    I was hoping they would be complelety separate from each other.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search