[metrics] Histogram samples never age out, leaving stale percentiles on low-frequency metrics
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
main (development)
### Please describe the bug 🐞
`DescriptiveStatisticsHistogram`, the only `Histogram` implementation used by Fluss metric groups (13 construction sites), keeps a fixed-size circular buffer with no time dimension: samples are evicted only by newer samples. On low-frequency metrics a short burst of slow samples keeps percentiles inflated for hours, and metrics that stop receiving samples never clear stale values until restart. The same implementation is present in v0.9.1-incubating.
Example: `logFlushLatencyMs` is updated only on infrequent flush paths — about 1.8 samples/min in one observed low-traffic deployment. In the deterministic 1024-sample reproduction, P99 stays at 2,000 ms and drops below 1 second only after 1,015 subsequent normal samples, about 9.4 hours at that rate, while `_count` keeps increasing — a P99 alert fires for hours although the server is healthy (verified on main @ f09de856b).
Expected: histogram samples age out by time, so a short anomaly fades within a bounded duration and idle metrics stop reporting stale values.
### Solution
Add a time bound to `DescriptiveStatisticsHistogram`: **current samples = last `maxSamples` ∩ last `maxAge`**.
- Keep a timestamp array next to the existing value ring; evict expired samples in both `update()` and `getStatistics()`, so scraping alone clears stale values even without new samples. `getCount()` stays cumulative.
- Add a `(maxSamples, maxAge)` constructor, keep the single-arg constructor unchanged, and migrate the 13 internal construction sites to a shared default `maxAge` of **5 minutes**, retaining existing caps (64/100/1024). At ~1.8 samples/min a 5-minute window holds ~9 samples; high-frequency metrics are still bounded by the sample cap.
- No new dependency; metric names, the `HistogramStatistics` contract, and the Prometheus export format stay unchanged. Semantics follow Kafka's `SampledStat` (event + time window); Dropwizard reservoirs were ruled out (no expiry on idle / no sample cap).
Out of scope: RocksDB native histogram gauges and Prometheus `_bucket` export changes.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with DescriptiveStatisticsHistogram and trace its 13 metric-group construction sites, noting the existing sample caps and the single-argument constructor. Review the update(), getStatistics(), and getCount() behavior, then verify that expired samples are removed during updates and reads while count remains cumulative. Done means the shared five-minute defaults are applied without changing metric names, the HistogramStatistics contract, or Prometheus output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 66/100