grafana / grafana/pyroscope

Compactor: pyroscope_compaction_size_bytes histogram buckets are too small

Open
#4,910 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

## Description

The `pyroscope_compaction_size_bytes` histogram has bucket boundaries that are far too small to capture actual compacted block sizes, making the metric effectively useless.

## Current bucket definition

https://github.com/grafana/pyroscope/blob/main/pkg/compactor/bucket_compactor.go#L290-L294

```go
m.Size = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "pyroscope_compaction_size_bytes",
Help: "Final block size after compaction by level",
Buckets: prometheus.ExponentialBuckets(32, 1.5, 12),
}, []string{"level"})
```

This generates buckets: `32, 48, 72, 108, 162, 243, 364, 546, 820, 1230, 1845, 2768` — maxing out at **~2.7 KB**.

## Problem

Compacted blocks are typically in the **MB to GB** range. Every observation lands in the `+Inf` bucket, so `histogram_quantile()` returns a flat value near the last finite bucket boundary (~3KB) regardless of actual block sizes.

```promql
-- Always returns ~3KB regardless of actual compaction sizes
histogram_quantile(0.95, sum(rate(pyroscope_compaction_size_bytes_bucket{cluster="$cluster"}[15m])) by (le))
```

## Suggested fix

Use buckets that cover the realistic range of compacted block sizes (MB to GB):

```go
Buckets: prometheus.ExponentialBuckets(1<<20, 2, 15),
// 1MB, 2MB, 4MB, 8MB, 16MB, 32MB, 64MB, 128MB, 256MB, 512MB, 1GB, 2GB, 4GB, 8GB, 16GB
```

Alternatively, this could be made configurable so operators can tune the bucket boundaries to match their workload characteristics.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.