linkerd / linkerd/linkerd2

The authority label in the proxy metrics has high cardinality in certain situations

Open
#5,746 8 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs/design pinned priority/P1
Dominant language
Go
Stars
11.5k
Forks
1.4k
Avg merge
3d 22h
Merged PRs (30d)
53

Description

## Bug Report

I was not sure whether to report this as a bug or a feature request. Please let me know whether I need to migrate the issue elsewhere.

### What is the issue?
The authority label in the proxy metrics has high cardinality values in certain situations.

### How can it be reproduced?
We have a SaaS product which handles customer traffic by using a subdomain for each tenant. We have 3000 customers at the moment which means the authority label from the proxy metrics has 3K distinct values. Coupled with the `target_addr` label, we easily get a dot product of 300K distinct time series. In addition, the latency metric, which is one of the golden signals, is a histogram with more than 10 buckets. This combination produces a total cardinality of >3M for just that one particular metric.

### Environment

- Kubernetes Version: 1.18
- Cluster Environment: EKS
- Linkerd version: 2.9.3

### Possible solution

In the ideal case, the metric labels from the proxy would be configurable, allowing the user to decide how granular they want to go.

Another solution would be to completely exclude the authority label from the proxy metrics.

### Additional context

For the time being, we decided run a custom build of the proxy with a hardcoded authority value in order to move forward with the implementation of linkerd. However, we are hoping that this issue can somehow be resolved so that we would avoid the overhead of maintaining a permanent fork.

In our situation, the authority label is really an application-specific piece of information that should be instrumented using a different system. It is fundamentally incompatible with Prometheus which is good at answering questions about the operational health of a particular service as a whole.

I also found an issue which raises a similar concern, but was closed at the time due to a lack of matching use cases: https://github.com/linkerd/linkerd2/issues/1378

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how Linkerd 2.x proxy metrics produce the authority and target_addr labels, using the reported high-cardinality scenario and related issue 1378 as context. Determine the supported configuration or exclusion behavior needed, and verify that the resulting metrics avoid the excessive cardinality described.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, prometheus
Domain
infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.