elastic / elastic/apm

Store upstream paths in transactions/spans for service maps

Open
#364 48 comments 7 reactions 0 assignees View on GitHub
Dominant language
Gherkin
Stars
427
Forks
125
PR merge metrics
No merged PRs in 30d

Description

We currently walk traces (via a scripted metric aggregation) to get paths/connections between services. However, that's untenable for a couple of reasons:

- We need to select traces to inspect first, and then walk the traces. This can be slow in many cases, and it's unpredictable.
- A scripted metric aggregation is a foot-gun, and it might be removed from the default distribution in the future, meaning we can no longer rely on it in the APM app.
- It depends on the presence of spans, meaning we can't just purely use (transaction, span) metrics to power the UI.

One solution is to store (hashed) paths in transaction or span metrics, per @axw's suggestion.

Here's how that could possibly work:

- Each service propagates a hash that uniquely identifies the service + the upstream path. Meaning, the root service A propagates a hash of just service A, service B propagates a hash of service A + service B, and so forth.
- This hash is propagated via the tracestate header.
- These hashes are also stored on transactions/spans (or the derived metrics).

We should consider the following use cases when deciding where and how to store the hashed paths:

- Global service maps
- Filtered service maps (e.g., by service name or environment)
- Dependency statistics (i.e., metrics for one service directly talking to another service/external dependency)

One requirement is that we should be able to resolve all connections with one or two requests, without using a scripted metric aggregation.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by resolving the proposed hash propagation and storage design against global and filtered service maps and dependency statistics; done means connections can be resolved in one or two requests without scripted metric aggregation.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.