monitoring: `/stats` field mappings drift across 3 upstream repos with no automated sync
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 543
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 109
Description
## The drift problem
When apm-server adds, renames, or removes a metric exposed by `apm-server:5066/stats`, five files in three other repositories need a matching update so the metric flows correctly through the stack-monitoring pipeline:
| File | Repo | Purpose |
| --- | --- | --- |
| [`monitoring-beats.json`](https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/template-resources/src/main/resources/monitoring-beats.json) | `elastic/elasticsearch` | Stack-monitoring index template (legacy) |
| [`monitoring-beats-mb.json`](https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/template-resources/src/main/resources/monitoring-beats-mb.json) | `elastic/elasticsearch` | Stack-monitoring index template (metricbeat variant) |
| [`metricbeat/module/beat/_meta/fields.yml`](https://github.com/elastic/beats/blob/main/metricbeat/module/beat/_meta/fields.yml) | `elastic/beats` | Alias entries used by metricbeat |
| [`metricbeat/module/beat/stats/_meta/fields.yml`](https://github.com/elastic/beats/blob/main/metricbeat/module/beat/stats/_meta/fields.yml) | `elastic/beats` | Concrete-typed entries under `beat.stats.*` |
| [`elastic_agent/data_stream/apm_server_metrics/fields/beat-stats-fields.yml`](https://github.com/elastic/integrations/blob/main/packages/elastic_agent/data_stream/apm_server_metrics/fields/beat-stats-fields.yml) | `elastic/integrations` | EA integration package fields |
Today this is a manual three-PR dance. There is no CI that fails when the trees diverge, so drift accumulates silently and is only discovered when a stack-monitoring chart breaks for a customer or when someone happens to look. Concrete examples of drift that has already accumulated:
- `apm-server.server.response.errors.concurrency` was removed from the server in #1749 (2019). It is still present in `monitoring-beats*.json`.
- `apm-server.jaeger.*` was removed in #14791. Still present in `monitoring-beats*.json`.
- `apm-server.sampling.tail.storage.disk_usage_threshold_pct` was added in #20464 (9.2.0+). The five mapping files have it; `/stats` did not actually emit it until #20993 / #20996, because of an unrelated translator gap.
- The full `apm-server.otlp.{grpc,http}.{logs,metrics,traces}.response.errors.*` family (closed/decode/forbidden/internal/invalidquery/method/notfound/queue/toolarge/unavailable/validate) is missing from the three downstream YAML files but appears in `/stats` post-#20993.
## Past attempt
[#13638](https://github.com/elastic/apm-server/pull/13638) added a Python script `script/stats_to_mapping.py` that read a single `/stats` snapshot and rewrote each of the five files in place. The PR was closed without merging, so the tool is not in `main` today. The script worked but lived in apm-server, which made running it require setting up a Python venv with `ruamel.yaml` — workable, but enough friction that nobody ran it on a regular cadence.
## Current state
- `/stats` is now an exhaustive enumeration of every metric apm-server defines, thanks to [#20993](https://github.com/elastic/apm-server/pull/20993) (eager monitoring counter registration). Before that PR, lazy OTel counters were missing from `/stats` until their code path fired, so any tooling that reads `/stats` saw a partial view.
- A Go port of the Python regen tool is being prepared as a separate PR against `elastic/apm-tools` (more appropriate home than apm-server, since it's developer tooling and the testdata goldens are ~10k+ lines).
- There is still no CI that detects drift between `/stats` and the five upstream files. The regen tool is part of the answer; it is not the whole answer.
## Proposal
Two complementary pieces:
1. **Regen tool (in progress, `elastic/apm-tools`)** — pipe `/stats` into the binary with the five file paths as arguments, and it rewrites the relevant subtrees in place. This is the active-developer workflow: when you add a metric, you run the tool, you open three PRs, you're done.
2. **Drift-detection CI job (this issue)** — a scheduled job (cron, e.g. weekly or on every apm-server release branch cut) that:
- starts a fresh apm-server build,
- captures `/stats`,
- clones the three upstream repos (or their `main` snapshots),
- runs the regen tool,
- fails (and ideally opens an issue or PR) if any of the five files would change.
This catches the silent-drift case: someone lands a metric change in apm-server without running the regen tool. Owning the job in apm-server (rather than relying on the upstream repos to detect it) is appropriate because apm-server is the source of truth.
The apm-tools PR fixes (1). It does not fix (2); that is the larger piece tracked here.
## Related
- Closed Python attempt: #13638
- Eager-counter fix that makes `/stats` a complete source: #20993
- Float-translator gap that hid one field from `/stats`: #20996
- Long-running upstream-mapping-sync work: #15533, #13475
Contributor guide
Assessment
This issue has not been assessed yet.