linkerd / linkerd/linkerd2

Support native histograms

Open
#15,373 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/viz enhancement
Dominant language
Go
Stars
11.5k
Forks
1.4k
Avg merge
3d 22h
Merged PRs (30d)
53

Description

### What problem are you trying to solve?

With each linkerd-proxy sidecar exporting histogram metrics, they're quickly becoming the dominant metrics in our Prometheus instances. There are also other benefits. Switching to native histograms (stable since Prometheus 3.8.0) would:
* decrease storage requirements,
* speed up various operations in Prometheus, including queries (due to huge cardinality drop),
* improve outlier reporting (native histograms define precision, not buckets).

See detailed comparison in [Prometheus docs](https://prometheus.io/docs/practices/histograms/).

### How should the problem be solved?

* Add a switch in Viz config that would change between native and classic histogram queries.
* Add native histograms to proxies and other components. There are at least two approaches possible.
* Always export both classic and native histograms (Prometheus has options to only scrape one of them).
* Add a config option (annotation in case of proxy?) that would allow selecting one of them.

Important note: native histograms only work with the protobuf encoding.

### Any alternatives you've considered?

It's possible to convert classic histograms losslessly to NHCB (Native Histograms with Custom Buckets) by tuning Prometheus scrape config. In this approach, no changes to exported metrics are required.

**Viz queries still need to be updated.**

### How would users interact with this feature?

Most likely a single boolean in the main helm chart and another one per extension. In case of Viz, it might either try to query both classic and native histograms or have a config option for that.

### Would you like to work on this feature?

maybe

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Viz config and queries, the main Helm chart and per-extension settings, and histogram export in linkerd-proxy and other components. Check the Prometheus protobuf-encoding constraint before choosing between dual export and a selection option. Done means native-histogram support is configurable, proxy and component exports are covered, and Viz queries handle the selected histogram type.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes, prometheus, rust
Domain
cloud, distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.