[Enhancement] Support suppressing minimal-value metrics to reduce export payload
- Dominant language
- Java
- Stars
- 22.6k
- Forks
- 12k
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
### Before Creating the Enhancement Request
- [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.
### Summary
This PR introduces an opt-in broker config `suppressMinValueMetrics` (default `false`, backward compatible) that suppresses exporting data points whose value is not greater than a
minimal threshold.
### Motivation
The consumer lag gauges are asynchronous ObservableGauges that emit one data point per consumer_group × topic × is_retry combination. In deployments with many consumer groups and topics, the vast majority of these combinations are idle at any given sampling instant, so their values are 0 — yet they are still recorded and exported on every collection cycle. This inflates both metric cardinality and the OTLP export payload.
In production, we observed zero-valued points accounting for ~95% of gauge data points and the bulk of the payload. Suppressing them is safe as long as the monitoring pipeline tolerates discontinuous series.
### Describe the Solution You'd Like
- `BrokerConfig`: add `suppressMinValueMetrics` (default `false`).
- `BrokerMetricsManager`: add a general guard `shouldRecordValue(currentValue, minValue)`.
When suppression is enabled, a data point is recorded only if `currentValue > minValue`.
- Apply the guard at the record site of the 5 consumer lag gauges, all passing `minValue = 0` for now, i.e. zero-valued points are dropped.
### Describe Alternatives You've Considered
.
### Additional Context
When enabled, threshold-based dashboards/alerts (e.g. `lag > N`) are unaffected; the only difference is
that idle series appear as gaps instead of a flat zero line. Therefore, before turning this on, make sure
there are **no alert rules based on `absent()` / `absent_over_time()`** (or any equivalent "no data" detection) targeting these metrics.
Contributor guide
Research direction
Start by locating BrokerConfig and BrokerMetricsManager, then find the five consumer lag gauge record sites described in the issue. Confirm the new option defaults to false and that enabling it suppresses values at or below the stated minimum while preserving existing behavior when disabled; also check the documented absent-series alert caveat.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100