skip to Main Content

Problem Summary

When using ingress nginx controller to route requests to my grpc server, the request metadata headers get stripped out. When I directly hit the service the headers are present.

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: <domain name>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: <service name>
            port:
              number: 50051
  tls:
  - hosts:
    - <domain name>
    secretName: <secret>

Troubleshooting I’ve done.

example command I used to request to nginx. When I inspect the server log, the "test.header" metadata is missing

./grpcurl -vv -cacert <cert path> -d '{<data>}' -H 'test.header:header' --proto <file> -import-path . <domain name>:443 <endpoint>

example command to request directly to the service. The "test.header" metadata comes through

./grpcurl -vv --plaintext -d '{<data>}' -H 'test.header:header' --proto <file> -import-path . <service name>:50051 <endpoint>

Inspecting nginx log with –v=5 gives me this. No error message, but the header is not there.

2022/08/26 02:54:34 [debug] 237#237: *20996 http2 table add: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 header: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 table add: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 header: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":method: POST"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":scheme: http"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":path: <path>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":authority: upstream_balancer"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-request-id: e89c7ad55ebd852a06416158be06f799"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-real-ip: <>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-for: <>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-host: <domain name>:443"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-port: 443"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-proto: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-scheme: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-scheme: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "te: trailers"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: 505249202a20485454502f322e300d0a0d0a534d0d0a0d0a00001204000000000000010000000000020000000000047fffffff0000040800000000007fff0000000136010400000001838644b16087aa534d6d83498f54482d9dcc42afdc2bde3d529a6b6c1a4c7ab716cee62158c454de3d529a6b6c1a4c7aaefacc5f5f418cb6b426c28e98a31d03a885b30089f2b585ed6950958d27972bcf91d1c8db658e479b10c0e341702dbd1940e4aebeff0087f2b5851d0b1abf87081765f5c4bbff008bf2b4a7b3c0ec90b22d29ec87081765f5c4bbff008cf2b4a7b3c0ec90b22d39d09f921d03498f51283db22e6a925b2a1d171a699f008cf2b4a7b3c0ec90b2..., len: 383
...

Looking through different documents, there seem to be some options for http

  1. "nginx.org/proxy-pass-headers" annotation
  2. "proxy_pass_request_headers" flag in nginx configuration: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_request_headers

But I’m struggling to find anything for grpc. The closest I could find was "grpc_set_header" flag in nginx configuration,

I’m very surprised and confused there isn’t many documents or posts about what I’m experiencing, which makes me think that I’m doing something wrong. Any help would be appreciated.

2

Answers


  1. Chosen as BEST ANSWER

    My issue was the dots in my header keys. https://trac.nginx.org/nginx/ticket/1558

    Nginx was ignoring the keys with dots, and when I removed the dots the headers started coming through. I also tried using the "ignore_invalid_headers: false" config as suggested in the Nginx ticket above, but that didn't work for me. Just had to update the header keys.


  2. Try with the in configmap use-forwarded-headers

    use-forwarded-headers: "true"

    Doc ref : https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers

    If your header container the underscore you can enable that also in configmap of Nginx ingress controller

    enable-underscores-in-headers: true

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search