elastic / elastic/apm

Traceparent header overriden by an intermediate service

Open
#609 12 comments 1 reaction 0 assignees View on GitHub
discussion
Dominant language
Gherkin
Stars
427
Forks
125
PR merge metrics
No merged PRs in 30d

Description

In the [W3C TraceContext](https://www.w3.org/TR/trace-context/) specification it's possible that a system is monitored by multiple tracing tools.

Example: **A** -> **B** -> **C** (Service **A** calls Service **B** which calls Service **C**). **A** and **C** are monitored by Elastic APM, B is monitored by another service.

Elastic APM is unable to link the transactions in **A** and **C**, since **C**'s parent is a transaction in **B** (since the third-party service in **B** writes a `traceparent` header), which Elastic APM doesn't know about since **B** is monitored by a third-party service.

If we control **B**, the problem is easy to fix:

- either monitor **B** with Elastic APM
- or don't monitor it and forward the `traceparent` header

However, **Google Cloud Run** does its own monitoring, which doesn't look like it can be disabled (in this case **A** is a downstream service, B is the Cloud Run runtime, and **C** is the service hosted in Cloud Run).

As a workaround, I have forked the elastic-apm-node agent and made it use the `elastic-apm-traceparent` header by default instead of the `traceparent` header. However I have no idea what a good long-term solution would be. Any ideas?

Related: https://github.com/elastic/apm/issues/286

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no repository files or tests. Start by reading the W3C TraceContext specification and tracing-header handling in the elastic-apm-node agent, then compare that behavior with Google Cloud Run's monitoring. Done would require an agreed long-term way to preserve transaction linkage across the intermediate service.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, node.js
Domain
distributed-systems, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.