How can I use an ngnix proxy to scrape Prometheus metrics using a custom HTTP header? – Nginx

I brought you a complete setup with an app, a forward proxy, and prometheus in docker-compose. It’s quite long, so I’m putting it after the explanation. Please note that, just as with your solution, it does not work with host.docker.internal as it seems that NGINX does not use /etc/hosts when resolving hosts: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/259#issuecomment-1125197753 . All other hosts should work fine and you can use host’s IP address instead of host.docker.internal if you need so.

You can run this by saving the contents into docker-compose.yml and running docker-compose up from the same directory. After 10 seconds or so you should see in logs how requests go through the proxy to the app and the app will show you the headers that it got. You can then proceed to Prometheus UI (localhost:9090) and query for metric the_answer_is to further check that everything is in place.

The proxy works as following:

• the target param (e.g. GET /?target=example.com) is the host or IP where the actual metrics are, this is the only mandatory parameter;
• if there is a scheme param, use it as the protocol, default – "http";
• if there is a host param, use it as Host header, default – the value of target param;
• if there is a port param, use it as a TCP port, default is "" (determined by the scheme);
• if there is a secret_token param, it gets injected into X-Custom-Header, default is "";

I recommend testing the proxy with curl, like this:

curl 'localhost/something/here?target=someapp&port=8000&secret_token=foo

Now goes the docker-compose.yml:

version: '3'
networks:
  app: # a network where the app is
  no_app: # a network where there is no app so that prometheus can't reach it directly

services:
  # A basic http server that exposes one metric and prints some headers along the way
  someapp:
   image: tiangolo/uwsgi-nginx-flask:python3.8-alpine
   entrypoint: ["/usr/local/bin/python", "-c"]
   networks:
    - app
   ports:
     - 8000:8000
   command:
   - |-
     import json
     from flask import Flask, request
     app = Flask(__name__)

     import logging
     log = logging.getLogger('werkzeug')

     @app.route('/', defaults={'path': ''})
     @app.route('/<path:path>')
     def print_request(path):
         log.info(f"{'-'*78}n{str(request.headers).strip()}")
         if request.path == "/something/here":
           return "the_answer_is 42n"
         else:
           return "OKn"

     app.run("0.0.0.0", port=8000)

  # A forward proxy for Prometheus
  forward_proxy:
    image: nginx
    networks:
      - app
      - no_app
    ports:
      - 80:80
    entrypoint: ["/bin/bash", "-c"]
    environment:
      # Note that single "$" is considered by docker-compose as its variable, 
      # double "$$" is just an escape here
      config: |
        # Default scheme
        map $$arg_scheme $$target_scheme {
          ~.+     $$arg_scheme;
          default http;
        }
        # Default host (header) from target
        map $$arg_host $$target_host {
          ~.+     $$arg_host;
          default $$arg_target;
        }
        # This is to add ":" between target ip or host and port
        map $$arg_port $$has_port {
          ~.+     ":";
          default "";
        }

        # use docker internal DNS to resolve "someapp"
        resolver 127.0.0.11 ipv6=off;

        server {
          listen 80;

          location / {
            proxy_set_header Host $$target_host;
            proxy_set_header X-Custom-Header $$arg_secret_token;
            proxy_pass $$target_scheme://$$arg_target$$has_port$$arg_port$$request_uri;
          }
        }

    command:
    - |-
      set -euo pipefail
      echo -e "$$config" >/etc/nginx/conf.d/default.conf
      echo -e "==== NGINX Config ====n$$(cat /etc/nginx/conf.d/default.conf)"
      nginx -g 'daemon off;'

  prometheus:
    image: prom/prometheus:v2.29.2
    entrypoint: ["/bin/sh", "-c"]
    ports:
      - 9090:9090
    command:
    - |
      echo -e "$$config" > /etc/prometheus/prometheus.yml
      echo -e "==== Prometheus Config ====n$$(cat /etc/prometheus/prometheus.yml)"
      /bin/prometheus --config.file=/etc/prometheus/prometheus.yml 
                      --storage.tsdb.path=/prometheus 
                      --web.console.libraries=/usr/share/prometheus/console_libraries 
                      --web.console.templates=/usr/share/prometheus/consoles
    networks:
      - no_app
    environment:
      config: |
        scrape_configs:
        - job_name: 'somejob'
          scrape_interval: 10s
          metrics_path: '/something/here'

          # set params for the proxy ($$arg_NAME)
          params:
            port: ["8000"]
            secret_token: ["foo"]  # beware, this will be visible in Prometheus UI under "config" section

          static_configs:
          - targets:
            - 'someapp'
          
          # Here we replace actual target with the address and port of our forward_proxy
          # If you're familiar with it, this is exactly the same as for blackbox exporter
          relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: forward_proxy:80  # The forward proxy address and port