I am trying to connect my pod from Kubernetes (k8s) cluster to a remote Jaeger server. I’ve tested and it can work well if both of them are on the same machine. However, when I run my app on k8s, my app can not connect to Jaeger despite I were using physical IP.
First, I’ve tried this:
containers:
- name: api
env:
- name: OTEL__AGENT_HOST
value: <my-physical-ip>
- name: OTEL__AGENT_PORT
value: "6831"
After read the docs from the internet, I add the Jaeger agent to my deployments as a sidecar container like this.
containers:
- name: api
env:
- name: OTEL__AGENT_HOST
value: "localhost"
- name: OTEL__AGENT_PORT
value: "6831"
- image: jaegertracing/jaeger-agent
name: jaeger-agent
ports:
- containerPort: 5775
protocol: UDP
- containerPort: 6831
protocol: UDP
- containerPort: 6832
protocol: UDP
- containerPort: 5778
protocol: TCP
args: ["--reporter.grpc.host-port=<my-physical-ip>:14250"]
It seems work very well on both containers. But on the collector of Jaeger, I received a log like this:
{"level":"warn","ts":1641987200.2678068,"caller":"channelz/logging.go:62","msg":"[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.
HandleStreams failed to receive the preface from client: read tcp 172.20.0.4:14250-><the-ip-of-machine-my-pods-are-deployed>:32852: i/o timeout"","system":"grpc","grpc_log":true}
2
Answers
I exposed port 14267 on Jaeger collector on remote machine, then change
args: ["--reporter.grpc.host-port=<my-physical-ip>:14250"]
toargs: ["--reporter.grpc.host-port=<my-physical-ip>:14267"]
and it works.Have you tried using jaeger operator? https://github.com/jaegertracing/jaeger-operator
This is how you will install it :
then you can create Jaeger instance that will up jaeger components like collector, agent, query . You can define storage too .. like elastic search for e.g.
Then in your application’s deployment yaml file you will need to configure agent as a side car (or u can use agent as deamonset) so that request can be forwarded to the collector ..
More details here: https://www.jaegertracing.io/docs/1.31/operator/#deployment-strategies