open-telemetry / open-telemetry/opentelemetry-cpp-contrib
NGINX: Allow overriding `opentelemetry_trace_id` from within the config
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 153
- Forks
- 184
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 26
Description
Is your feature request related to a problem?
We use Google Load Balancers which add a vendor specific header (X-Cloud-Trace-Context) to our requests. It seems compatible with W3C format, but OTEL library does not detect it as it's not supported and creates a new span which gets passed down the line (together with Google header), resulting in two separate traces which are not connected.
To solve it, we tried is parsing the X-Cloud-Trace-Context header within NGINX and extracting trace ID and using that to construct a new header which is passed down the line:
http {
# omitted: some parsing code to $trace_id from X-Cloud-Trace-Context header
server {
opentelemetry_propagate;
opentelemetry_trust_incoming_spans on;
location / {
proxy_pass http://some_endpoint;
# Construct a custom header using Google's trace ID and OTEL's generated Span ID
set $custom_traceparent "00-${trace_id}-${opentelemetry_span_id}-01";
# Set header
proxy_set_header traceparent $custom_traceparent;
}
}
}
This solution passes a valid trace down the line, however it also results in two traces due to OTEL library generating it's own opentelemetry_trace_id which it then sends to the OLTP exporter host. It's not entirely clear to me at what point in the NGINX request flow OTEL generates its trace ID and I'm wondering if there is a way to override it. This can be visualised as follows:
graph TD
A(Request) -->B(Google Load Balancer)
B --> |Google adds: X-Cloud-Trace-Context|C(NGiNX)
C -->|Custom traceparent - Trace A| D(Application)
C -->|Default/Internal traceparent - Trace B| E(OLTP exporter host)
Describe the solution you'd like
- Approach 1: Ideal solution would be to find out if there's a way to generate and set headers within NGINX before the
OtelGetTraceIdfunction gets called (code). - Approach 2: Alternatively, since NGINX does not seem to support overriding already set variables, perhaps the library functionality could be extended to allow overriding this value from within the code.
Thanks and I hope you can provide some additional context how to best solve it!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with instrumentation/nginx/src/otel_ngx_module.cpp at the linked OtelGetTraceId function, then trace when the NGINX module generates and exports its trace ID relative to configuration processing. Done means the configuration can use the Google-derived trace ID consistently for propagation and OLTP export without creating a second trace.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, nginx
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100