elastic / elastic/observability-migration-platform

[metric_map] Emit Class-2 entries (transform / attribute_filter / unit_scale) instead of recording gaps

Open
#314 1 comment 0 reactions 0 assignees View on GitHub
asset:dashboards capabilities quality source:datadog source:grafana workstream:translator
Dominant language
Python
Stars
6
Forks
8
Avg merge
2d 22h
Merged PRs (30d)
23

Description

## Problem statement

The shared metric map (PR #305, on `main`) classifies each entry via `classify_metric_map_entry` in `observability_migration/core/metric_mapping/entries.py`:

- **Class-1 (`exact`)**: pure `source_metric -> target_field` rename. Applied and emitted (`schema.py resolve_metric_field` returns the mapped target, winning over profile/passthrough).
- **Class-2 (`requires_transform`)**: any entry with `transform != none` (`to_rate` / `drop_rate`), a non-empty `attribute_filter`, or `unit_scale != 1.0`.

Today Class-2 entries are parsed and validated, then **recorded as a gap/warning and not emitted** (`resolve_metric_field` does not apply a bare rename for them; it appends to `metric_map_gaps` / `metric_map_warnings`). This is correct degrade-gracefully behavior (better a flagged gap than silently wrong math), but it means the common existing-OTel scenarios still do not translate:

- Attribute-split metrics: separate source metrics collapsing into one target field plus an attribute (e.g. `container_network_receive_bytes_total` -> `k8s.pod.network.io` with `attribute_filter: {network.direction: receive}`).
- Rate/counter semantics: `to_rate` / `drop_rate` when the target stores a different temporality than the source.
- Unit conversions: `unit_scale` (e.g. nanocores to cores, bytes to KiB).

## Proposed change

Make the ES|QL translator actually emit Class-2 entries instead of only flagging them:

1. `attribute_filter` -> add the corresponding `WHERE`/filter predicate on the target attribute(s) when binding the mapped field.
2. `transform` (`to_rate` / `drop_rate`) -> reconcile with the existing counter-aware rate path (native `RATE()`/`INCREASE()` where field-caps prove counter, else the documented bucket-span approximation), so a `to_rate` mapping produces the right rate and `drop_rate` strips a redundant one.
3. `unit_scale` -> multiply/scale the emitted expression by the factor.
4. Preserve degrade-gracefully: where correctness cannot be proven (e.g. temporality unknown without `_field_caps`), keep the gap/warning rather than emitting wrong math, and record applied-vs-gap in the contract artifacts.

Source-neutral: this benefits both Grafana (attribute-split k8s metrics) and Datadog (Agent to OTel rate/unit changes).

## Code pointers

- `observability_migration/core/metric_mapping/entries.py` (`classify_metric_map_entry`, `VALID_TRANSFORMS`).
- `observability_migration/adapters/source/grafana/schema.py` (`resolve_metric_field`, `metric_map_gaps` / `metric_map_warnings` / `metric_map_applied`).
- Datadog counter/rate path and `unit_scale` ergonomics.

## Related

Follows PR #305. Related: #308 (Datadog rate/temporality/unit correctness), #309 (Grafana metric-name mapping, attribute-split + per-metric data stream), #311 (multi-target overlays incl. attribute-split / cross-data-stream).

Contributor guide

Open the contributing guide

Research direction

Start with classify_metric_map_entry and VALID_TRANSFORMS in observability_migration/core/metric_mapping/entries.py, then trace resolve_metric_field and the metric_map_gaps, metric_map_warnings, and metric_map_applied paths in the Grafana schema adapter. Read the Datadog counter/rate path and related issues for existing semantics. Done means Class-2 mappings are applied only when correctness is proven, otherwise remain recorded as gaps or warnings with applied-vs-gap contract artifacts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.