elastic / elastic/apm-server

monitoring: Tail-based sampling `disk_usage_threshold_pct` stat silently dropped from /stats

Open
#20,996 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

**APM Server version** (`apm-server version`): 9.2.0+, 9.3.0+.

**Description of the problem including expected versus actual behavior**:

The tail-based sampling storage metric `apm-server.sampling.tail.storage.disk_usage_threshold_pct` is recorded by the storage manager as a `Float64Gauge`, but it never appears in the `/stats` HTTP endpoint output even when TBS is enabled and the gauge is being actively recorded.

The metric was introduced in #20464 (backports: #20477 to 9.2, #20478 to 9.3). It has been silently dropped since it was added — it has never been observable via `/stats`.

Root cause: in `internal/beatcmd/beat.go`, the OTel-to-libbeat translator that walks `apm-server.*` instruments (`addAPMServerMetricsToMap`) only handles `Sum[int64]` / `Gauge[int64]` aggregations. Any float64 instrument falls through and is silently dropped:

```go
// before
if v, ok := getScalarInt64(m.Data); ok {
beatsMetrics[m.Name] = v
}
// no float64 branch — float-typed instruments under apm-server.* vanish
```

The int64-only translator was introduced as part of the OTel monitoring migration in #15094. `disk_usage_threshold_pct` is the only `apm-server.*` Float64 instrument today, so it's the only one missing from `/stats`. But the gap is general: any future float-typed `apm-server.*` metric (e.g. ratios, percentages, durations recorded as seconds) would hit the same trap.

Expected: when TBS is enabled, `apm-server.sampling.tail.storage.disk_usage_threshold_pct` should appear in `/stats` with the configured threshold value (e.g. `0.8`).

Actual: the field is absent regardless of configuration.

**Steps to reproduce**:

1. Run apm-server 9.2.0+ with tail-based sampling enabled and a configured `apm-server.sampling.tail.storage_limit` / disk threshold.
2. Curl the stats endpoint: `curl localhost:5066/stats | jq '.apm-server.sampling.tail.storage'`
3. Note that `disk_usage_threshold_pct` is missing, while sibling int64 fields (e.g. `lsm_size`, `value_log_size`) are present.

**Provide logs (if relevant)**:

N/A — silent drop, no log output.

---

This was discovered while fixing the lazy-counter regression in #20993; the fix for this specific issue is included there (extend the translator to try `getScalarFloat64` when the int64 path fails). Filing this as a separate issue so the regression is tracked independently.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.