elastic / elastic/observability-migration-platform
[metric_map] Emit Class-2 entries (transform / attribute_filter / unit_scale) instead of recording gaps
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 23
Description
## Problem statement
The shared metric map (PR #305, on `main`) classifies each entry via `classify_metric_map_entry` in `observability_migration/core/metric_mapping/entries.py`:
- **Class-1 (`exact`)**: pure `source_metric -> target_field` rename. Applied and emitted (`schema.py resolve_metric_field` returns the mapped target, winning over profile/passthrough).
- **Class-2 (`requires_transform`)**: any entry with `transform != none` (`to_rate` / `drop_rate`), a non-empty `attribute_filter`, or `unit_scale != 1.0`.
Today Class-2 entries are parsed and validated, then **recorded as a gap/warning and not emitted** (`resolve_metric_field` does not apply a bare rename for them; it appends to `metric_map_gaps` / `metric_map_warnings`). This is correct degrade-gracefully behavior (better a flagged gap than silently wrong math), but it means the common existing-OTel scenarios still do not translate:
- Attribute-split metrics: separate source metrics collapsing into one target field plus an attribute (e.g. `container_network_receive_bytes_total` -> `k8s.pod.network.io` with `attribute_filter: {network.direction: receive}`).
- Rate/counter semantics: `to_rate` / `drop_rate` when the target stores a different temporality than the source.
- Unit conversions: `unit_scale` (e.g. nanocores to cores, bytes to KiB).
## Proposed change
Make the ES|QL translator actually emit Class-2 entries instead of only flagging them:
1. `attribute_filter` -> add the corresponding `WHERE`/filter predicate on the target attribute(s) when binding the mapped field.
2. `transform` (`to_rate` / `drop_rate`) -> reconcile with the existing counter-aware rate path (native `RATE()`/`INCREASE()` where field-caps prove counter, else the documented bucket-span approximation), so a `to_rate` mapping produces the right rate and `drop_rate` strips a redundant one.
3. `unit_scale` -> multiply/scale the emitted expression by the factor.
4. Preserve degrade-gracefully: where correctness cannot be proven (e.g. temporality unknown without `_field_caps`), keep the gap/warning rather than emitting wrong math, and record applied-vs-gap in the contract artifacts.
Source-neutral: this benefits both Grafana (attribute-split k8s metrics) and Datadog (Agent to OTel rate/unit changes).
## Code pointers
- `observability_migration/core/metric_mapping/entries.py` (`classify_metric_map_entry`, `VALID_TRANSFORMS`).
- `observability_migration/adapters/source/grafana/schema.py` (`resolve_metric_field`, `metric_map_gaps` / `metric_map_warnings` / `metric_map_applied`).
- Datadog counter/rate path and `unit_scale` ergonomics.
## Related
Follows PR #305. Related: #308 (Datadog rate/temporality/unit correctness), #309 (Grafana metric-name mapping, attribute-split + per-metric data stream), #311 (multi-target overlays incl. attribute-split / cross-data-stream).
Contributor guide
Research direction
Start with classify_metric_map_entry and VALID_TRANSFORMS in observability_migration/core/metric_mapping/entries.py, then trace resolve_metric_field and the metric_map_gaps, metric_map_warnings, and metric_map_applied paths in the Grafana schema adapter. Read the Datadog counter/rate path and related issues for existing semantics. Done means Class-2 mappings are applied only when correctness is proven, otherwise remain recorded as gaps or warnings with applied-vs-gap contract artifacts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100