elastic / elastic/apm-server

monitoring: `/stats` field mappings drift across 3 upstream repos with no automated sync

Open
#21,000 0 comments 0 reactions 1 assignee Claimed by @carsonip View on GitHub
enhancement
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

## The drift problem

When apm-server adds, renames, or removes a metric exposed by `apm-server:5066/stats`, five files in three other repositories need a matching update so the metric flows correctly through the stack-monitoring pipeline:

| File | Repo | Purpose |
| --- | --- | --- |
| [`monitoring-beats.json`](https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/template-resources/src/main/resources/monitoring-beats.json) | `elastic/elasticsearch` | Stack-monitoring index template (legacy) |
| [`monitoring-beats-mb.json`](https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/template-resources/src/main/resources/monitoring-beats-mb.json) | `elastic/elasticsearch` | Stack-monitoring index template (metricbeat variant) |
| [`metricbeat/module/beat/_meta/fields.yml`](https://github.com/elastic/beats/blob/main/metricbeat/module/beat/_meta/fields.yml) | `elastic/beats` | Alias entries used by metricbeat |
| [`metricbeat/module/beat/stats/_meta/fields.yml`](https://github.com/elastic/beats/blob/main/metricbeat/module/beat/stats/_meta/fields.yml) | `elastic/beats` | Concrete-typed entries under `beat.stats.*` |
| [`elastic_agent/data_stream/apm_server_metrics/fields/beat-stats-fields.yml`](https://github.com/elastic/integrations/blob/main/packages/elastic_agent/data_stream/apm_server_metrics/fields/beat-stats-fields.yml) | `elastic/integrations` | EA integration package fields |

Today this is a manual three-PR dance. There is no CI that fails when the trees diverge, so drift accumulates silently and is only discovered when a stack-monitoring chart breaks for a customer or when someone happens to look. Concrete examples of drift that has already accumulated:

- `apm-server.server.response.errors.concurrency` was removed from the server in #1749 (2019). It is still present in `monitoring-beats*.json`.
- `apm-server.jaeger.*` was removed in #14791. Still present in `monitoring-beats*.json`.
- `apm-server.sampling.tail.storage.disk_usage_threshold_pct` was added in #20464 (9.2.0+). The five mapping files have it; `/stats` did not actually emit it until #20993 / #20996, because of an unrelated translator gap.
- The full `apm-server.otlp.{grpc,http}.{logs,metrics,traces}.response.errors.*` family (closed/decode/forbidden/internal/invalidquery/method/notfound/queue/toolarge/unavailable/validate) is missing from the three downstream YAML files but appears in `/stats` post-#20993.

## Past attempt

[#13638](https://github.com/elastic/apm-server/pull/13638) added a Python script `script/stats_to_mapping.py` that read a single `/stats` snapshot and rewrote each of the five files in place. The PR was closed without merging, so the tool is not in `main` today. The script worked but lived in apm-server, which made running it require setting up a Python venv with `ruamel.yaml` — workable, but enough friction that nobody ran it on a regular cadence.

## Current state

- `/stats` is now an exhaustive enumeration of every metric apm-server defines, thanks to [#20993](https://github.com/elastic/apm-server/pull/20993) (eager monitoring counter registration). Before that PR, lazy OTel counters were missing from `/stats` until their code path fired, so any tooling that reads `/stats` saw a partial view.
- A Go port of the Python regen tool is being prepared as a separate PR against `elastic/apm-tools` (more appropriate home than apm-server, since it's developer tooling and the testdata goldens are ~10k+ lines).
- There is still no CI that detects drift between `/stats` and the five upstream files. The regen tool is part of the answer; it is not the whole answer.

## Proposal

Two complementary pieces:

1. **Regen tool (in progress, `elastic/apm-tools`)** — pipe `/stats` into the binary with the five file paths as arguments, and it rewrites the relevant subtrees in place. This is the active-developer workflow: when you add a metric, you run the tool, you open three PRs, you're done.

2. **Drift-detection CI job (this issue)** — a scheduled job (cron, e.g. weekly or on every apm-server release branch cut) that:
- starts a fresh apm-server build,
- captures `/stats`,
- clones the three upstream repos (or their `main` snapshots),
- runs the regen tool,
- fails (and ideally opens an issue or PR) if any of the five files would change.

This catches the silent-drift case: someone lands a metric change in apm-server without running the regen tool. Owning the job in apm-server (rather than relying on the upstream repos to detect it) is appropriate because apm-server is the source of truth.

The apm-tools PR fixes (1). It does not fix (2); that is the larger piece tracked here.

## Related

- Closed Python attempt: #13638
- Eager-counter fix that makes `/stats` a complete source: #20993
- Float-translator gap that hid one field from `/stats`: #20996
- Long-running upstream-mapping-sync work: #15533, #13475

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.