linkerd / linkerd/linkerd2

Add source labels to inbound metrics

Open
#4,101 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

rfc
Dominant language
Go
Stars
11.5k
Forks
1.4k
Avg merge
3d 22h
Merged PRs (30d)
53

Description

## Feature Request

### What problem are you trying to solve?

I would like to answer the question:
For a given proxy/pod/workload, what are the inbound metrics, grouped by client?

Additionally, per the [Prometheus docs](https://prometheus.io/docs/instrumenting/writing_exporters/#labels), a single metric should always be exported with the same set of labels. We currently do not satisfy this requirement (e.g. `dst_deployment` is only present on `outbound` metrics).

### How should the problem be solved?

For `inbound` metrics, introduce new labels:
- `src_deployment` (new data)
- `src_namespace` (new data)
- `dst_deployment` (copy of `deployment`)
- `dst_namespace` (copy of `namespace`)

For `outbound` metrics, introduce new labels:
- `src_deployment` (copy of `deployment`)
- `src_namespace` (copy of `namespace`)

Notes:
- Enables grouping by client, when looking at a single proxy.
- Makes the set of labels consistent for a given metric.
- Backwards compatible with existing Linkerd tooling.
- Applies to `request_total` and `response_total` metrics.
- Applies to workload types (`statefulset`, `daemonset`, etc).

#### Current example

```
response_total{
direction="inbound",
namespace="linkerd"
deployment="linkerd-controller",
}
response_total{
direction="outbound",
namespace="linkerd"
deployment="linkerd-web",
dst_namespace="linkerd",
dst_deployment="linkerd-controller",
}
```

#### Proposed example

```
response_total{
direction="inbound",
src_namespace="linkerd"
src_deployment="linkerd-web",
dst_namespace="linkerd"
dst_deployment="linkerd-controller",
}
response_total{
direction="outbound",
src_namespace="linkerd"
src_deployment="linkerd-web",
dst_namespace="linkerd",
dst_deployment="linkerd-controller",
}
```

### Any alternatives you've considered?

Today, we can query `outbound` metrics, with `dst_` labels set to what we want to measure inbound for:

```
sum(
response_total{
direction="outbound"
dst_namespace="linkerd",
dst_deployment="linkerd-controller",
}
) by (deployment, namespace)
```

Drawbacks to this approach:
- This query must be done against Prometheus. This data is not attainable from a single Linkerd proxy.
- It's difficult to reconcile these metrics with the `inbound` metrics of the proxy or workload we are interested in.

### How would users interact with this feature?

Existing Linkerd CLIs / UI would not need to change, but this new data would allow simplification of `linkerd stat` queries in the CLI, dashboard, and Grafana. Users would also have better visibility into an unhealthy pod/workload's client behavior.

Relates to #4102.

/cc @adleong @grampelberg

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named. Start by locating the request_total and response_total metric definitions and the code supplying inbound and outbound workload labels; verify the proposed labels are present consistently for both directions and workload types, with tests covering the resulting label sets.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, prometheus
Domain
backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.