The authority label in the proxy metrics has high cardinality in certain situations
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 11.5k
- Forks
- 1.4k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 53
Description
## Bug Report
I was not sure whether to report this as a bug or a feature request. Please let me know whether I need to migrate the issue elsewhere.
### What is the issue?
The authority label in the proxy metrics has high cardinality values in certain situations.
### How can it be reproduced?
We have a SaaS product which handles customer traffic by using a subdomain for each tenant. We have 3000 customers at the moment which means the authority label from the proxy metrics has 3K distinct values. Coupled with the `target_addr` label, we easily get a dot product of 300K distinct time series. In addition, the latency metric, which is one of the golden signals, is a histogram with more than 10 buckets. This combination produces a total cardinality of >3M for just that one particular metric.
### Environment
- Kubernetes Version: 1.18
- Cluster Environment: EKS
- Linkerd version: 2.9.3
### Possible solution
In the ideal case, the metric labels from the proxy would be configurable, allowing the user to decide how granular they want to go.
Another solution would be to completely exclude the authority label from the proxy metrics.
### Additional context
For the time being, we decided run a custom build of the proxy with a hardcoded authority value in order to move forward with the implementation of linkerd. However, we are hoping that this issue can somehow be resolved so that we would avoid the overhead of maintaining a permanent fork.
In our situation, the authority label is really an application-specific piece of information that should be instrumented using a different system. It is fundamentally incompatible with Prometheus which is good at answering questions about the operational health of a particular service as a whole.
I also found an issue which raises a similar concern, but was closed at the time due to a lack of matching use cases: https://github.com/linkerd/linkerd2/issues/1378
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how Linkerd 2.x proxy metrics produce the authority and target_addr labels, using the reported high-cardinality scenario and related issue 1378 as context. Determine the supported configuration or exclusion behavior needed, and verify that the resulting metrics avoid the excessive cardinality described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, prometheus
- Domain
- infrastructure, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100