elastic / elastic/observability-migration-platform

[epic] metric_map completeness: source metrics → existing OTel data in Elasticsearch

Open
#320 0 comments 0 reactions 0 assignees View on GitHub
asset:dashboards capabilities program:grafana-engine quality source:grafana workstream:translator
Dominant language
Python
Stars
6
Forks
8
Avg merge
2d 22h
Merged PRs (30d)
23

Description

## Why this epic

Migrating Grafana (and Datadog) dashboards onto **existing OTel data already in Elasticsearch** fails on vocabulary, not on the query engine. Source dashboards use Prometheus/cAdvisor (or Datadog) metric names; Elastic stores dotted OTel semantic-convention fields via OTLP. Bridging them is the `metric_map` program started by #305.

### Live evidence (origin/main @ 5da0dd2, ES seeded with real OTel k8s fields)

Real Grafana **Kubernetes / Compute Resources / Pod** dashboard (16 panels), `--field-profile otel --es-url --preflight`:

- **Baseline, no map: 0/16 panels.** Every metric `missing` in `required_target_contract.json` even though the equivalent dotted OTel field exists (57 fields discovered).
- **With map, Class-1 renames applied:** `container_memory_working_set_bytes → container.memory.working_set` and `container_memory_rss → container.memory.rss` flipped to `confirmed`.
- **Class-2 not applied:** `container_network_{receive,transmit}_bytes_total → k8s.pod.network.io {direction}` stayed `missing`; recorded as a gap, `k8s.pod.network.io` never emitted.
- **Gaps not surfaced:** `metric_map_gaps()/warnings()` are populated but no artifact reads them.
- **Recording rules (5) and cross-data-stream limits/requests** have no single-rename target.

## Scope / sub-issues

- #309 — metric-name mapping layer: Class-1 (done via #305), a shipped default map, and the **otlptranslator forward-index auto-suggest** generator. (Data-stream targeting moved to #311.)
- #314 — emit **Class-2** entries (`attribute_filter` → WHERE, `transform` → rate, `unit_scale` → scaling); until then, **surface** gaps/warnings in report + panel notes.
- #311 — multi-series overlays incl. attribute-split and **cross-data-stream / per-metric target data stream**.
- #310 — detect Prometheus **recording rules** (no target field) and flag instead of empty panels.
- #315 — apply `metric_map` in the **native PromQL** path (currently bypassed).

## Related (separate source track)

- #308 — Datadog rate/temporality/unit correctness (same Class-2 machinery, Datadog source).

Contributor guide

Open the contributing guide

Research direction

Treat this as an epic and start by reading the listed sub-issues, especially #309 and #314. Inspect metric_map, required_target_contract.json, and the metric_map_gaps()/warnings() entry points, then trace the relevant dashboard migration path. Done means the selected sub-issue’s mapping behavior and reporting gaps are covered without leaving the broader epic scope ambiguous.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, grafana, prometheus, python
Domain
data-engineering, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.