Spans emitted by linkerd-proxy make it hard to use SpanMetrics due to spanName cardinality explosion
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 11.5k
- Forks
- 1.4k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 53
Description
### What problem are you trying to solve?
According to [Jaeger docs](https://www.jaegertracing.io/docs/1.45/spm/), when adding SpanMetrics you can expect the metrics cardinality to be:
```
max = 324 * num_operations
typical = 72 * num_operations
```
However, by default spanNames as emitted by linkerd-proxy have a full URL path set (which is a bad practice, [according to Opentelemetry specs](https://github.com/open-telemetry/opentelemetry-specification/issues/270)), which results in the following situation, when URL path contains variables (pretty common for APIs):
It is possible to rename spanNames, and extract variables to attributes, by using Opentelemetry Collector's SpanProcessor, for example in case above:
```
processors:
...
span/to_attributes:
name:
to_attributes:
rules:
- ^\/api\/v1\/users\/(?P.*)\/own_permissions\/(?P.*)
...
service:
...
pipelines:
traces:
processors: [ batch,span/to_attributes ]
```
But that would mean you have to hardcode all the paths of all APIs that have variables in URL paths, or devise of some mechanism to automatically update the central Collector's config based on what individual services have/use.
I'm really curious how do other people use SpanMetrics when using linkerd. Do you just live with this issue and cardinality explosion? Is there some other config I'm missing, or some extension/project that deals with this? As far as I can see, linkerd-proxy specifically makes SpanMetrics really hard to use.
### How should the problem be solved?
I have no idea what the alternative to using full URL path as spanName would be, considering Linkerd (probably?) doesn't know which parts of the path are fixed, and which are variable.
### Any alternatives you've considered?
I gave the OpentelemetryCollector SpanProcessor config above, but like I said, I would have to hardcode the pattern of _every_ such API call. Doesn't seem very maintainable.
### How would users interact with this feature?
_No response_
### Would you like to work on this feature?
None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing how linkerd-proxy emits span names and the Jaeger SpanMetrics and OpenTelemetry Collector references in the issue. Define a supported approach that avoids unbounded URL-path cardinality without hardcoding every API route, then validate it with focused tracing tests and documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, rust
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100