skip to Main Content

I’m managing Kubernetes + nginx.

I’d like to install dynamic modules on nginx that are provided by Nginx Ingress Controller.
Those dynamic modules are not offered by Nginx Ingress Controller official configmap (https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/)

So I believe, I need to build my own Docker container of Nginx Ingress Controller.
(Could be added at this? https://github.com/kubernetes/ingress-nginx/blob/8951b7e22ad3952c549150f61d7346f272c563e1/images/nginx/rootfs/build.sh#L618-L632 )

Do you know how we can customize the controller and manage it by helm chart? I’m thinking about making a Fork branch from the controller master repo on Github.
But I don’t have any idea on how we install a customized version of the controller on terraform + helm chart.

However, I would prefer to use a non-customizable solution (because of some annotation settings)

Environment:
Kubernetes
Nginx Ingress Controller is installed by helm chart + terraform
Nginx Ingress Controller -> https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx

Terraform:

resource "helm_release" "nginx-ingress-controller" {
  name      = "nginx-ingress-controller"
  chart     = "ingress-nginx/ingress-nginx"
  namespace = "kube-system"
  version   = "3.34.0"
}

dynamic modules
https://docs.nginx.com/nginx/admin-guide/dynamic-modules/dynamic-modules/
(install process might be using --add-dynamic-module option, and set load_module modules/something.so on nginx.conf via ingress.yaml)

Thank you.

2

Answers


  1. Please take a look at Cloud Native Buildpacks.
    Images can be built directly from application source without additional instructions.

    Maybe this nginx-buildpack solves your problem:

    Loading dynamic modules
    You can use templates to set the path to a dynamic module using the load_module directive.

    PS. https://12factor.net/build-release-run

    The twelve-factor app uses strict separation between the build, release, and run stages. For example, it is impossible to make changes to the code at runtime, since there is no way to propagate those changes back to the build stage.

    Login or Signup to reply.
  2. TL;DR

    Extend the official image with the dynamic modules, and update the helm_release terraform resource to set the controller.image.registry, controller.image.image, controller.image.tag, controller.image.digest, and controller.image.digestChroot for your custom image along with a controller.config.main-snippet to load the dynamic module(s) in the main context.


    This is similar to my previous answer for building modules using the official nginx image. You can extend the ingress-nginx/controller image, build the modules in one stage, extend the official image with the dynamic modules in another stage, and use the image in your helm_release. An example for extending the ingress-nginx/controller with the echo-nginx-module e.g.:

    Docker

    ARG INGRESS_NGINX_CONTROLLER_VERSION
    FROM registry.k8s.io/ingress-nginx/controller:${INGRESS_NGINX_CONTROLLER_VERSION} as build
    
    ARG INGRESS_NGINX_CONTROLLER_VERSION
    ENV INGRESS_NGINX_CONTROLLER_VERSION=${INGRESS_NGINX_CONTROLLER_VERSION}
    
    USER root
    RUN apk add 
            automake 
            ca-certificates 
            curl 
            gcc 
            g++ 
            make 
            pcre-dev 
            zlib-dev
    
    RUN NGINX_VERSION=$(nginx -V 2>&1 |sed -n -e 's/nginx version: //p' |cut -d'/' -f2); 
        curl -L "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" | tar -C /tmp/nginx --strip-components=1 -xz
    
    WORKDIR /src/echo-nginx-module
    RUN curl -L https://github.com/openresty/echo-nginx-module/archive/refs/tags/v0.63.tar.gz | tar --strip-components=1 -xz
    
    WORKDIR /tmp/nginx
    RUN ./configure --with-compat --add-dynamic-module=/src/echo-nginx-module && 
        make modules
    
    FROM registry.k8s.io/ingress-nginx/controller:${INGRESS_NGINX_CONTROLLER_VERSION}
    
    COPY --from=build /tmp/nginx/objs/ngx_http_echo_module.so /etc/nginx/modules/
    

    … build and push the image e.g.: docker build --rm -t myrepo/ingress-nginx/controller:v1.5.1-echo --build-arg INGRESS_NGINX_CONTROLLER_VERSION=v1.5.1 . && docker push myrepo/ingress-nginx/controller:v1.5.1-echo

    Terraform

    Update the terraform helm_release resource to install the charts using the custom image and adding a main-snippet to set the load_module directive in the main context:

    resource "helm_release" "ingress-nginx" {
      name       = "ingress-nginx"
      namespace  = "kube-system"
      repository = "https://kubernetes.github.io/ingress-nginx"
      chart      = "ingress-nginx"
      version    = "3.34.0"
      set {
        name = "controller.image.registry"
        value = "myrepo"
      }
      set {
        name = "controller.image.image"
        value = "ingress-nginx/controller"
      }
      set {
        name = "controller.image.tag"
        value = "v1.5.1-echo"
      }
      set {
        name = "controller.image.digest"
        value = "sha256:1b32b3e8c983ef4a32d87dead51fbbf2a2c085f1deff6aa27a212ca6beefcb72"
      }
      set {
        name = "controller.image.digestChroot"
        value = "sha256:f2e1146adeadac8eebb251284f45f8569beef9c6ec834ae1335d26617da6af2d"
      }
      set {
        name = "controller.config.main-snippet"
        value = <<EOF
    load_module /etc/nginx/modules/ngx_http_echo_module.so;
    EOF
      }
    }
    

    The controller.image.digest is the image RepoDigest: docker inspect myrepo/ingress-nginx/controller:v1.5.1-echo --format '{{range .RepoDigests}}{{println .}}{{end}}' |cut -d'@' -f2

    The controller.image.digestChroot is the Parent sha: docker inspect myrepo/ingress-nginx/controller:v1.5.1-echo --format {{.Parent}}

    Test

    1. Create a nginx pod: kubectl run nginx --image=nginx
    2. Expose the pod: kubectl expose pod nginx --port 80 --target-port 80
    3. Create an ingress with a server-snippet:
    cat <<EOF | kubectl apply -f -
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: nginx
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-prod
        nginx.ingress.kubernetes.io/server-snippet: |
          location /hello {
            echo "hello, world!";
          }
        kubernetes.io/ingress.class: nginx
    spec:
      rules:
      - host: echo.example.com
        http:
          paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: nginx
                port:
                  number: 80
      tls:
      - hosts:
        - echo.example.com
        secretName: tls-echo
    EOF
    

    Using cert-manager for TLS certificates issuance and external-dns for DNS management.

    1. Test using curl:

    curl echo module test on publicly exposed app

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search