elastic / elastic/observability-migration-platform
[epic] metric_map completeness: source metrics → existing OTel data in Elasticsearch
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 23
Description
## Why this epic
Migrating Grafana (and Datadog) dashboards onto **existing OTel data already in Elasticsearch** fails on vocabulary, not on the query engine. Source dashboards use Prometheus/cAdvisor (or Datadog) metric names; Elastic stores dotted OTel semantic-convention fields via OTLP. Bridging them is the `metric_map` program started by #305.
### Live evidence (origin/main @ 5da0dd2, ES seeded with real OTel k8s fields)
Real Grafana **Kubernetes / Compute Resources / Pod** dashboard (16 panels), `--field-profile otel --es-url --preflight`:
- **Baseline, no map: 0/16 panels.** Every metric `missing` in `required_target_contract.json` even though the equivalent dotted OTel field exists (57 fields discovered).
- **With map, Class-1 renames applied:** `container_memory_working_set_bytes → container.memory.working_set` and `container_memory_rss → container.memory.rss` flipped to `confirmed`.
- **Class-2 not applied:** `container_network_{receive,transmit}_bytes_total → k8s.pod.network.io {direction}` stayed `missing`; recorded as a gap, `k8s.pod.network.io` never emitted.
- **Gaps not surfaced:** `metric_map_gaps()/warnings()` are populated but no artifact reads them.
- **Recording rules (5) and cross-data-stream limits/requests** have no single-rename target.
## Scope / sub-issues
- #309 — metric-name mapping layer: Class-1 (done via #305), a shipped default map, and the **otlptranslator forward-index auto-suggest** generator. (Data-stream targeting moved to #311.)
- #314 — emit **Class-2** entries (`attribute_filter` → WHERE, `transform` → rate, `unit_scale` → scaling); until then, **surface** gaps/warnings in report + panel notes.
- #311 — multi-series overlays incl. attribute-split and **cross-data-stream / per-metric target data stream**.
- #310 — detect Prometheus **recording rules** (no target field) and flag instead of empty panels.
- #315 — apply `metric_map` in the **native PromQL** path (currently bypassed).
## Related (separate source track)
- #308 — Datadog rate/temporality/unit correctness (same Class-2 machinery, Datadog source).
Contributor guide
Research direction
Treat this as an epic and start by reading the listed sub-issues, especially #309 and #314. Inspect metric_map, required_target_contract.json, and the metric_map_gaps()/warnings() entry points, then trace the relevant dashboard migration path. Done means the selected sub-issue’s mapping behavior and reporting gaps are covered without leaving the broader epic scope ambiguous.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, grafana, prometheus, python
- Domain
- data-engineering, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100