apache / apache/fluss

[metrics] Histogram samples never age out, leaving stale percentiles on low-frequency metrics

Open
#4,139 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

main (development)

### Please describe the bug 🐞

`DescriptiveStatisticsHistogram`, the only `Histogram` implementation used by Fluss metric groups (13 construction sites), keeps a fixed-size circular buffer with no time dimension: samples are evicted only by newer samples. On low-frequency metrics a short burst of slow samples keeps percentiles inflated for hours, and metrics that stop receiving samples never clear stale values until restart. The same implementation is present in v0.9.1-incubating.

Example: `logFlushLatencyMs` is updated only on infrequent flush paths — about 1.8 samples/min in one observed low-traffic deployment. In the deterministic 1024-sample reproduction, P99 stays at 2,000 ms and drops below 1 second only after 1,015 subsequent normal samples, about 9.4 hours at that rate, while `_count` keeps increasing — a P99 alert fires for hours although the server is healthy (verified on main @ f09de856b).

Expected: histogram samples age out by time, so a short anomaly fades within a bounded duration and idle metrics stop reporting stale values.

### Solution

Add a time bound to `DescriptiveStatisticsHistogram`: **current samples = last `maxSamples` ∩ last `maxAge`**.

- Keep a timestamp array next to the existing value ring; evict expired samples in both `update()` and `getStatistics()`, so scraping alone clears stale values even without new samples. `getCount()` stays cumulative.
- Add a `(maxSamples, maxAge)` constructor, keep the single-arg constructor unchanged, and migrate the 13 internal construction sites to a shared default `maxAge` of **5 minutes**, retaining existing caps (64/100/1024). At ~1.8 samples/min a 5-minute window holds ~9 samples; high-frequency metrics are still bounded by the sample cap.
- No new dependency; metric names, the `HistogramStatistics` contract, and the Prometheus export format stay unchanged. Semantics follow Kafka's `SampledStat` (event + time window); Dropwizard reservoirs were ruled out (no expiry on idle / no sample cap).

Out of scope: RocksDB native histogram gauges and Prometheus `_bucket` export changes.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with DescriptiveStatisticsHistogram and trace its 13 metric-group construction sites, noting the existing sample caps and the single-argument constructor. Review the update(), getStatistics(), and getCount() behavior, then verify that expired samples are removed during updates and reads while count remains cumulative. Done means the shared five-minute defaults are applied without changing metric names, the HistogramStatistics contract, or Prometheus output.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
66/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.