Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Ubuntu – Prometheus not able to read metrics from otel collector in docker

I am trying to setup a docker-compose that will have prometheus, otel-collector and a spring boot application that will send data to the otel collector.

The docker compose file is like this:

version: '3.4'
networks:
  cart:
    driver: bridge

services:
  prometheus:
    build: prometheus
    ports:
      - "9090:9090"
    networks:
      - cart


  otel-collector:
    build: otel-collector
    networks:
      - cart
    ports:
      - "1888:1888" # pprof extension
      - "8888:8888" # Prometheus metrics exposed by the Collector
      - "8889:8889" # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317" # OTLP gRPC receiver
      - "4318:4318" # OTLP http receiver
      - "55679:55679" # zpages extension

My otel collector configuration file is this:

receivers:
  otlp:
    protocols:
      http:

processors:
  # batch metrics before sending to reduce API usage
  batch:

exporters:
  prometheus:
      endpoint: 0.0.0.0:8889
      namespace: default

# https://github.com/open-telemetry/opentelemetry-collector/blob/main/extension/README.md
extensions:
  # responsible for responding to health check calls on behalf of the collector.
  health_check:
  # fetches the collector’s performance data
  pprof:
  # serves as an http endpoint that provides live debugging data about instrumented components.
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

and my prometheus configuration like this:

global:
  scrape_interval: 10s # How frequently to scrape targets by default
  evaluation_interval: 10s
  
scrape_configs:
  - job_name: 'otel-collector'   # The job name is assigned to scraped metrics by default.
    static_configs:                       # A static_config allows specifying a list of targets and a common label set for them
      - targets: ["otel-collector:8889"]

The dockerfile for the otel collector is this:

FROM otel/opentelemetry-collector-contrib:0.100.0

ADD otel-collector-config.yml /etc/otelcol-contrib/config.yml

and for prometheus:

FROM ubuntu/prometheus:2.50.0-22.04_stable

ADD prometheus.yml /etc/prometheus/prometheus.yml

But prometheus does not seem to be able to connect to the otel collector.

enter image description here

Which is the right way to configure prometheus to connect to the otel collector?

Tags:

2

Answers


  1. 0

    As per this doc

    I want to connect from a container to a service on the host
    The host has a changing IP address, or none if you have no network access. We recommend that you connect to the special DNS name host.docker.internal, which resolves to the internal IP address used by the host.

    I would recommend trying out that if you can set prometheus scrape target to point to host.docker.internal:8889 instead of otel-collector:8889, since prometheus might not able to resolve that DNS name.

    Login or Signup to reply.
  2. 0

    You have a mistake in your Dockerfile. Image otel/opentelemetry-collector-contrib:0.100.0 use /etc/otelcol-contrib/config.yaml.

    So otel/opentelemetry-collector-contrib:0.100.0 is using the default config.yaml file.
    Metrics are available by 8888 port.

    Image otel/opentelemetry-collector-contrib:0.100.0 use /etc/otelcol-contrib/config.yaml

    Solution 1

    You need change in Otel’s Dockerfile the next line

    ADD otel-collector-config.yml /etc/otelcol-contrib/config.yml
    -> ADD otel-collector-config.yml /etc/otelcol-contrib/config.yaml

    Solution 2

    Change the target in the Prometheus config otel-collector:8889 to otel-collector:8888

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search