skip to Main Content

I am trying to connect my pod from Kubernetes (k8s) cluster to a remote Jaeger server. I’ve tested and it can work well if both of them are on the same machine. However, when I run my app on k8s, my app can not connect to Jaeger despite I were using physical IP.
First, I’ve tried this:

containers:
 - name: api
   env:
    - name: OTEL__AGENT_HOST
      value: <my-physical-ip>
    - name: OTEL__AGENT_PORT
      value: "6831"

After read the docs from the internet, I add the Jaeger agent to my deployments as a sidecar container like this.

containers:
        - name: api
          env:
            - name: OTEL__AGENT_HOST
              value: "localhost"
            - name: OTEL__AGENT_PORT
              value: "6831"
        - image: jaegertracing/jaeger-agent
          name: jaeger-agent
          ports:
          - containerPort: 5775
            protocol: UDP
          - containerPort: 6831
            protocol: UDP
          - containerPort: 6832
            protocol: UDP
          - containerPort: 5778
            protocol: TCP
          args: ["--reporter.grpc.host-port=<my-physical-ip>:14250"]

It seems work very well on both containers. But on the collector of Jaeger, I received a log like this:

{"level":"warn","ts":1641987200.2678068,"caller":"channelz/logging.go:62","msg":"[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.
HandleStreams failed to receive the preface from client: read tcp 172.20.0.4:14250-><the-ip-of-machine-my-pods-are-deployed>:32852: i/o timeout"","system":"grpc","grpc_log":true}

2

Answers


  1. Chosen as BEST ANSWER

    I exposed port 14267 on Jaeger collector on remote machine, then change args: ["--reporter.grpc.host-port=<my-physical-ip>:14250"] to args: ["--reporter.grpc.host-port=<my-physical-ip>:14267"] and it works.


  2. Have you tried using jaeger operator? https://github.com/jaegertracing/jaeger-operator

    This is how you will install it :

    kubectl create namespace observability 
    kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.31.0/jaeger-operator.yaml -n observability
    

    then you can create Jaeger instance that will up jaeger components like collector, agent, query . You can define storage too .. like elastic search for e.g.

    apiVersion: jaegertracing.io/v1
    kind: Jaeger
    metadata:
      name: simple-prod-es
    spec:
      strategy: production
      storage:
        type: elasticsearch 
        options:
          es:
            server-urls: https://search-test-g7fbo7pzghdquvvgxty2pc6lqu.us-east-2.es.amazonaws.com
            index-prefix: jaeger-span
            username: test
            password: xxxeee
    

    Then in your application’s deployment yaml file you will need to configure agent as a side car (or u can use agent as deamonset) so that request can be forwarded to the collector ..

    More details here: https://www.jaegertracing.io/docs/1.31/operator/#deployment-strategies

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search