Traceparent header overriden by an intermediate service
- Dominant language
- Gherkin
- Stars
- 427
- Forks
- 125
- PR merge metrics
- No merged PRs in 30d
Description
In the [W3C TraceContext](https://www.w3.org/TR/trace-context/) specification it's possible that a system is monitored by multiple tracing tools.
Example: **A** -> **B** -> **C** (Service **A** calls Service **B** which calls Service **C**). **A** and **C** are monitored by Elastic APM, B is monitored by another service.
Elastic APM is unable to link the transactions in **A** and **C**, since **C**'s parent is a transaction in **B** (since the third-party service in **B** writes a `traceparent` header), which Elastic APM doesn't know about since **B** is monitored by a third-party service.
If we control **B**, the problem is easy to fix:
- either monitor **B** with Elastic APM
- or don't monitor it and forward the `traceparent` header
However, **Google Cloud Run** does its own monitoring, which doesn't look like it can be disabled (in this case **A** is a downstream service, B is the Cloud Run runtime, and **C** is the service hosted in Cloud Run).
As a workaround, I have forked the elastic-apm-node agent and made it use the `elastic-apm-traceparent` header by default instead of the `traceparent` header. However I have no idea what a good long-term solution would be. Any ideas?
Related: https://github.com/elastic/apm/issues/286
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no repository files or tests. Start by reading the W3C TraceContext specification and tracing-header handling in the elastic-apm-node agent, then compare that behavior with Google Cloud Run's monitoring. Done would require an agreed long-term way to preserve transaction linkage across the intermediate service.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, node.js
- Domain
- distributed-systems, observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100