Ingress nginx stripping grpc request metadata headers

Problem Summary

When using ingress nginx controller to route requests to my grpc server, the request metadata headers get stripped out. When I directly hit the service the headers are present.

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: <domain name>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: <service name>
            port:
              number: 50051
  tls:
  - hosts:
    - <domain name>
    secretName: <secret>

Troubleshooting I’ve done.

example command I used to request to nginx. When I inspect the server log, the "test.header" metadata is missing

./grpcurl -vv -cacert <cert path> -d '{<data>}' -H 'test.header:header' --proto <file> -import-path . <domain name>:443 <endpoint>

example command to request directly to the service. The "test.header" metadata comes through

./grpcurl -vv --plaintext -d '{<data>}' -H 'test.header:header' --proto <file> -import-path . <service name>:50051 <endpoint>

Inspecting nginx log with –v=5 gives me this. No error message, but the header is not there.

2022/08/26 02:54:34 [debug] 237#237: *20996 http2 table add: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 header: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 table add: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 header: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":method: POST"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":scheme: http"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":path: <path>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":authority: upstream_balancer"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-request-id: e89c7ad55ebd852a06416158be06f799"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-real-ip: <>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-for: <>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-host: <domain name>:443"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-port: 443"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-proto: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-scheme: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-scheme: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "te: trailers"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: 505249202a20485454502f322e300d0a0d0a534d0d0a0d0a00001204000000000000010000000000020000000000047fffffff0000040800000000007fff0000000136010400000001838644b16087aa534d6d83498f54482d9dcc42afdc2bde3d529a6b6c1a4c7ab716cee62158c454de3d529a6b6c1a4c7aaefacc5f5f418cb6b426c28e98a31d03a885b30089f2b585ed6950958d27972bcf91d1c8db658e479b10c0e341702dbd1940e4aebeff0087f2b5851d0b1abf87081765f5c4bbff008bf2b4a7b3c0ec90b22d29ec87081765f5c4bbff008cf2b4a7b3c0ec90b22d39d09f921d03498f51283db22e6a925b2a1d171a699f008cf2b4a7b3c0ec90b2..., len: 383
...

Looking through different documents, there seem to be some options for http

1. "nginx.org/proxy-pass-headers" annotation
2. "proxy_pass_request_headers" flag in nginx configuration: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_request_headers

But I’m struggling to find anything for grpc. The closest I could find was "grpc_set_header" flag in nginx configuration,

• http://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_set_header
• https://github.com/kubernetes/ingress-nginx/pull/2524
  but I’m not sure if I can set it to the values from the request with this flag, and ideally, I want to just forward all headers from my requests.

I’m very surprised and confused there isn’t many documents or posts about what I’m experiencing, which makes me think that I’m doing something wrong. Any help would be appreciated.