I’m pretty new to OpenTelemetry world.
We have a PHP backend and Node.js Express server for SSR. For starters we want to collect tracing data for some subset of the requests. So, when a request comes to PHP server it calculates some chances, and if it is true it starts collecting tracing data from PHP side. Then comes to a point that it makes a HTTP request to Express server to render some JS and HTML. And the option of if the tracing is enabled or not is passed to Express server endpoint via PHP.
The problem:
Tracing is setup since the beginning of the Express server, to track everything correctly. But the option to stop tracing or not, comes from backend to Express server in a route. So, when the route is invoked I need to check the body and decide to stop tracing or not.
I can’t do it with samplers. Because like I said the decision to not send any tracing comes to the endpoint. Not since the beginning.
How can I stop the tracing and don’t send anything? Even if some data were already collected.
2
Answers
In OpenTelemetry, the signal between services on whether to trace or not is called Context Propagation and should be propagated between services (in your case, from PHP to Express) via the W3C
traceparent
header [1].This header contains the parent trace id, span id, flags which indicate whether the parent is sampled or not. This is how traces across your two different services can be linked together (by having the same trace id, and having the root Express span linked to the PHP span which initiated the http request).
What you are describing, where PHP makes the decision to trace (via sampling), and Express should only trace based on PHP’s decision, can be handled by a "Parent-based, always-off" sampler in Express, and probably a "Parent-based, trace id ratio-based" sampler in PHP. You won’t need to discard any data from Express, since it will only trace if the parent was traced.
Lastly, you need to ensure that your outbound HTTP requests from PHP inject the appropriate traceparent header:
[1] https://www.w3.org/TR/trace-context/#traceparent-header
Disabling OpenTelemetry tracing after it has been initialized is not a straightforward task because OpenTelemetry is designed to be initialized and configured at the start of an application and remain consistent throughout its lifetime. However, you can manage the behavior of tracing dynamically to some extent using different strategies.
Here’s a general approach you could take to conditionally enable or disable tracing based on a flag received from your PHP backend:
Use a Dynamic Flag: Instead of trying to disable OpenTelemetry, you can check the flag in each request handler and decide whether or not to create a new span for tracing. This is not true disabling but rather conditional tracing.
Middleware to Act on the Flag: Implement middleware in your Express server that checks the tracing flag in the incoming request from the PHP server and conditionally creates spans based on this flag.
Suppress Instrumentation: You can suppress or modify span generation dynamically, using a custom span processor that acts based on the request properties or global flags.
Here’s how you could implement the above approach using Node.js and Express:
In the above code, ConditionalExportSpanProcessor is a custom span processor that uses a dynamic function shouldTrace to determine whether or not to export a span. The middleware updates this condition based on the request’s tracingEnabled flag. The shouldTrace function will need to be defined by you and can include any logic you need to decide whether to trace or not.
This strategy allows you to maintain the state of tracing throughout your application’s runtime while providing the flexibility to enable or disable it based on the needs of individual requests. It’s important to note that this method will only prevent the export of new spans created after the flag is received and won’t retroactively stop spans that have already been started from being exported.