`pyroscope_segment_writer_client_sent_bytes` metric has unbounded cardinality
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
#### Describe the bug
The cardinality of the `pyroscope_segment_writer_client_sent_bytes` metric can grow without bounds. The total number of metrics emitted by the `/metrics` endpoint requires ever-increasing amounts of memory to process from standard collectors (like otel-collector).
The histogram metric has dimensions on `shard`, `tenant`, and `addr`.
There is no code for clearing / deleting old tracked series from the metric.
As downstream segment-writer Pods restart, they can get a new IP address, creating new series for `addr`. The old series under the now-unused IP address linger.
If tenants are spread across many shards and/or adaptive placement is making significant changes to things, we can also see orphaned series as a result.
#### To Reproduce
Default behavior. Restart some segment-writer Pods and watch series count of `pyroscope_segment_writer_client_sent_bytes` grow.
#### Expected behavior
Metrics series shouldn't grow without bound.
How you achieve this is up for debate.
But it usually entails a TTL or similar orphan pruning mechanism on metrics series.
For the github.com/prometheus/client_golang/prometheus Go module, there are `DeleteLabelValues()`, `Delete()`, `DeletePartialMatch()`, and `Reset()` methods to remove series.
There is already precedence for this pattern in the Pyroscope code base. See e.g. https://github.com/grafana/pyroscope/blob/b4d74078918e396eaffa30273b913d18f9c0a42c/pkg/scheduler/scheduler.go#L622
#### Environment
k8s
#### Additional Context
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue through the /metrics endpoint while restarting segment-writer Pods and watching the series count for pyroscope_segment_writer_client_sent_bytes. Read pkg/scheduler/scheduler.go around line 622 for the existing metric-series cleanup pattern, then review the Prometheus DeleteLabelValues(), Delete(), DeletePartialMatch(), and Reset() methods. Done means old shard, tenant, and addr series no longer accumulate without bound.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100