Send the trace data of a website using Jaeger and Opentelemetry to Opensearch – WordPress

I’m working on the observability part of Opensearch so I’m trying to collect the trace data of a wordpress website and send it to Opensearch.

I’m collecting the trace data using the wordpress plugin Decalog, this later sends the data to Jaeger agent, then from jaeger i’m sending the data to Opentelemetry and then to Data prepper and lastly to Opensearch.

• Jaeger agent service in docker-compose :

  jaeger-agent:
    container_name: jaeger-agent
    image: jaegertracing/jaeger-agent:latest
    command: [ "--reporter.grpc.host-port=otel-collector:14250" ]
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778/tcp"
    networks:
      - our-network

The "command" ligne got me this error : Err: connection error: desc = "transport: Error while dialing dial tcp: lookup otel-collector on 127.0.0.11:53: server misbehaving"","system":"grpc","grpc_log":true
So I changed otel-collector to the IP of the otel-collector container.

• Otel collector and data prepper are installed using docker-compose.

  data-prepper:
    restart: unless-stopped
    container_name: data-prepper
    image: opensearchproject/data-prepper:latest
    volumes:
      - ./data-prepper/examples/trace_analytics_no_ssl.yml:/usr/share/data-prepper/pipelines.yaml
      - ./data-prepper/examples/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml
      - ./data-prepper/examples/demo/root-ca.pem:/usr/share/data-prepper/root-ca.pem
    ports:
      - "21890:21890"
    networks:
      - our-network
    depends_on:
      - "opensearch"
  otel-collector:
    container_name: otel-collector
    image: otel/opentelemetry-collector:0.54.0
    command: [ "--config=/etc/otel-collector-config.yml" ]
    working_dir: "/project"
    volumes:
      - ${PWD}/:/project
      - ./otel-collector-config.yml:/etc/otel-collector-config.yml
      - ./data-prepper/examples/demo/demo-data-prepper.crt:/etc/demo-data-prepper.crt
    ports:
      - "4317:4317"
    depends_on:
      - data-prepper
    networks:
      - our-network

The configuration of otel.yaml (to send data from opentelemetry to opensearch):

receivers:
  jaeger:
    protocols:
      grpc:

exporters:
  otlp/2:
    endpoint: data-prepper:21890
    tls:
      insecure: true
      insecure_skip_verify: true
  logging:

service:
  pipelines:
    traces:
      receivers: [jaeger]
      exporters: [logging, otlp/2]

The configuration for data prepper pipeline : entry-pipeline:

  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
    - otel_trace_raw_prepper:
  sink:
    - opensearch:
        hosts: [ "http://localhost:9200" ]
        cert: "/usr/share/data-prepper/root-ca.pem"
        username: "admin"
        password: "admin"
        trace_analytics_raw: true
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: ["http://localhost:9200"]
        cert: "/usr/share/data-prepper/root-ca.pem"
        username: "admin"
        password: "admin"
        trace_analytics_service_map: true

As of now I’m getting the following errors:

Jaeger agent :

Err: connection error: desc = "transport: Error while dialing dial tcp otel-collector-container-IP:14250: i/o timeout"","system":"grpc","grpc_log":true}

Open telemetry collector :

2022-08-04T15:31:32.675Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-08-04T15:31:32.675Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-08-04T15:31:32.675Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "otlp/2"}
2022-08-04T15:31:32.682Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "traces", "name": "otlp/2"}
2022-08-04T15:31:32.682Z        info    pipelines/pipelines.go:86       Starting processors...
2022-08-04T15:31:32.682Z        info    pipelines/pipelines.go:98       Starting receivers...
2022-08-04T15:31:32.682Z        info    pipelines/pipelines.go:102      Exporter is starting... {"kind": "receiver", "name": "jaeger", "pipeline": "traces"}
2022-08-04T15:31:32.683Z        info    static/strategy_store.go:203    No sampling strategies provided or URL is unavailable, using defaults   {"kind": "receiver", "name": "jaeger", "pipeline": "traces"}
2022-08-04T15:31:32.683Z        info    pipelines/pipelines.go:106      Exporter started.       {"kind": "receiver", "name": "jaeger", "pipeline": "traces"}
2022-08-04T15:31:32.683Z        info    service/collector.go:220        Starting otelcol...     {"Version": "0.54.0", "NumCPU": 2}
2022-08-04T15:31:32.683Z        info    service/collector.go:128        Everything is ready. Begin running and processing data.
2022-08-04T15:31:32.684Z        warn    zapgrpc/zapgrpc.go:191  [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "data-prepper:21890",
  "ServerName": "data-prepper:21890",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp data-prepper-container-ip:21890: connect: connection refused"        {"grpc_log": true}

Data prepper :

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.amazon.dataprepper.DataPrepper]: Constructor threw exception; nested exception is java.lang.RuntimeException: No valid pipeline is available for execution, exiting

Followed by this at the end :

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2022-08-04T15:23:22,803 [main] INFO  com.amazon.dataprepper.parser.config.DataPrepperAppConfiguration - Command line args: /usr/share/data-prepper/pipelines.yaml,/usr/share/data-prepper/data-prepper-config.yaml
2022-08-04T15:23:22,806 [main] INFO  com.amazon.dataprepper.parser.config.DataPrepperArgs - Using /usr/share/data-prepper/pipelines.yaml configuration file