letsencrypt / letsencrypt/boulder

bdns: reduce cardinality of histograms

Open
#6,142 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.8k
Forks
649
Avg merge
3d 23h
Merged PRs (30d)
24

Description

Using the query from https://www.robustperception.io/dropping-metrics-at-scrape-time-with-prometheus: topk(20, count by (__name__, job)({__name__=~".+"})), I found that the histograms in bdns are far and away our highest cardinality metrics.

image

That's because they are histograms (already high cardinality), and have many labels, and some of the label values have high cardinality (for instance, the resolver).

We should tweak these: for the histogram metrics, we are mostly interested in the time to do various operations, so we should remove almost all labels. We can then put those labels on counter metrics, which inherently have lower cardinality. This will reduce our storage and memory requirements for Prometheus, and will make querying these DNS histograms cheaper.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the bdns histogram and counter metric definitions, then review their labels and the Prometheus cardinality query linked in the issue. Done means histogram metrics retain only labels needed for operation timing, while the required labels move to lower-cardinality counters and the resulting metric cardinality is reduced.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.