Add source labels to inbound metrics
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 11.5k
- Forks
- 1.4k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 53
Description
## Feature Request
### What problem are you trying to solve?
I would like to answer the question:
For a given proxy/pod/workload, what are the inbound metrics, grouped by client?
Additionally, per the [Prometheus docs](https://prometheus.io/docs/instrumenting/writing_exporters/#labels), a single metric should always be exported with the same set of labels. We currently do not satisfy this requirement (e.g. `dst_deployment` is only present on `outbound` metrics).
### How should the problem be solved?
For `inbound` metrics, introduce new labels:
- `src_deployment` (new data)
- `src_namespace` (new data)
- `dst_deployment` (copy of `deployment`)
- `dst_namespace` (copy of `namespace`)
For `outbound` metrics, introduce new labels:
- `src_deployment` (copy of `deployment`)
- `src_namespace` (copy of `namespace`)
Notes:
- Enables grouping by client, when looking at a single proxy.
- Makes the set of labels consistent for a given metric.
- Backwards compatible with existing Linkerd tooling.
- Applies to `request_total` and `response_total` metrics.
- Applies to workload types (`statefulset`, `daemonset`, etc).
#### Current example
```
response_total{
direction="inbound",
namespace="linkerd"
deployment="linkerd-controller",
}
response_total{
direction="outbound",
namespace="linkerd"
deployment="linkerd-web",
dst_namespace="linkerd",
dst_deployment="linkerd-controller",
}
```
#### Proposed example
```
response_total{
direction="inbound",
src_namespace="linkerd"
src_deployment="linkerd-web",
dst_namespace="linkerd"
dst_deployment="linkerd-controller",
}
response_total{
direction="outbound",
src_namespace="linkerd"
src_deployment="linkerd-web",
dst_namespace="linkerd",
dst_deployment="linkerd-controller",
}
```
### Any alternatives you've considered?
Today, we can query `outbound` metrics, with `dst_` labels set to what we want to measure inbound for:
```
sum(
response_total{
direction="outbound"
dst_namespace="linkerd",
dst_deployment="linkerd-controller",
}
) by (deployment, namespace)
```
Drawbacks to this approach:
- This query must be done against Prometheus. This data is not attainable from a single Linkerd proxy.
- It's difficult to reconcile these metrics with the `inbound` metrics of the proxy or workload we are interested in.
### How would users interact with this feature?
Existing Linkerd CLIs / UI would not need to change, but this new data would allow simplification of `linkerd stat` queries in the CLI, dashboard, and Grafana. Users would also have better visibility into an unhealthy pod/workload's client behavior.
Relates to #4102.
/cc @adleong @grampelberg
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named. Start by locating the request_total and response_total metric definitions and the code supplying inbound and outbound workload labels; verify the proposed labels are present consistently for both directions and workload types, with tests covering the resulting label sets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, prometheus
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100