monitoring: Tail-based sampling `disk_usage_threshold_pct` stat silently dropped from /stats
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 543
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 109
Description
**APM Server version** (`apm-server version`): 9.2.0+, 9.3.0+.
**Description of the problem including expected versus actual behavior**:
The tail-based sampling storage metric `apm-server.sampling.tail.storage.disk_usage_threshold_pct` is recorded by the storage manager as a `Float64Gauge`, but it never appears in the `/stats` HTTP endpoint output even when TBS is enabled and the gauge is being actively recorded.
The metric was introduced in #20464 (backports: #20477 to 9.2, #20478 to 9.3). It has been silently dropped since it was added — it has never been observable via `/stats`.
Root cause: in `internal/beatcmd/beat.go`, the OTel-to-libbeat translator that walks `apm-server.*` instruments (`addAPMServerMetricsToMap`) only handles `Sum[int64]` / `Gauge[int64]` aggregations. Any float64 instrument falls through and is silently dropped:
```go
// before
if v, ok := getScalarInt64(m.Data); ok {
beatsMetrics[m.Name] = v
}
// no float64 branch — float-typed instruments under apm-server.* vanish
```
The int64-only translator was introduced as part of the OTel monitoring migration in #15094. `disk_usage_threshold_pct` is the only `apm-server.*` Float64 instrument today, so it's the only one missing from `/stats`. But the gap is general: any future float-typed `apm-server.*` metric (e.g. ratios, percentages, durations recorded as seconds) would hit the same trap.
Expected: when TBS is enabled, `apm-server.sampling.tail.storage.disk_usage_threshold_pct` should appear in `/stats` with the configured threshold value (e.g. `0.8`).
Actual: the field is absent regardless of configuration.
**Steps to reproduce**:
1. Run apm-server 9.2.0+ with tail-based sampling enabled and a configured `apm-server.sampling.tail.storage_limit` / disk threshold.
2. Curl the stats endpoint: `curl localhost:5066/stats | jq '.apm-server.sampling.tail.storage'`
3. Note that `disk_usage_threshold_pct` is missing, while sibling int64 fields (e.g. `lsm_size`, `value_log_size`) are present.
**Provide logs (if relevant)**:
N/A — silent drop, no log output.
---
This was discovered while fixing the lazy-counter regression in #20993; the fix for this specific issue is included there (extend the translator to try `getScalarFloat64` when the int64 path fails). Filing this as a separate issue so the regression is tracked independently.
Contributor guide
Assessment
This issue has not been assessed yet.