linkerd / linkerd/linkerd2

Reduce Prometheus Data Retention Duration

Open
#3,997 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

good first issue help wanted
Dominant language
Go
Stars
11.5k
Forks
1.4k
Avg merge
3d 22h
Merged PRs (30d)
53

Description

Following a [recent slack report](https://linkerd.slack.com/archives/C89RTCWJF/p1580228200108300) on a Prometheus OOMKilled incident, we want to revisit the effect of reducing the `storage.tsdb.retention.time` duration from the current default of 6 hours to 1 hour. The previous investigation, per https://github.com/linkerd/linkerd2/issues/3258#issuecomment-526377964, showed that it did help to reduce the memory footprint. The obvious tradeoff is that there will be less metrics data available to the user.

It was also reported that reducing the `storage.tsdb.min-block-duration` to 30 mins and `storage.tsdb.max-block-duration` to 1 hour, from the default 2 hours, reduced Prometheus' memory usage by less than half. Note that these are [hidden flags](https://github.com/prometheus/prometheus/blob/9c67fce6e0123e3acc85c6ec4cc89e79cb2010d8/cmd/prometheus/main.go#L186-L188), marked for "testing only". We may or may not want to rely on them, but it's still worth investigating.

Open Questions:

* Shall we introduce make these settings configurable via CLI flags, config file (see https://github.com/linkerd/linkerd2/issues/3870), K8s configmap etc.?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the Prometheus flags in cmd/prometheus/main.go linked by the issue, then read the related discussions in issues #3258 and #3870. Determine whether and where Linkerd should expose retention and block-duration settings, and define completion around a decided configuration path, validated defaults, and the documented metrics-retention tradeoff.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, prometheus
Domain
infrastructure, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.