elastic / elastic/observability-migration-platform

Investigate: histogram component-series (_count/_sum) references produce empty panels on OTel targets

Open
#271 0 comments 0 reactions 0 assignees View on GitHub
asset:dashboards program:grafana-engine quality source:grafana workstream:translator
Dominant language
Python
Stars
6
Forks
8
Avg merge
2d 22h
Merged PRs (30d)
23

Description

## Context

Surfaced in a customer Grafana→Kibana migration. The histogram `kubelet_pod_start_duration_seconds` produced a panel whose migrated ES|QL is:

```
FROM metrics-prometheusreceiver.otel-*
| WHERE kubelet_active_pods IS NOT NULL AND kubelet_pod_start_duration_seconds_count IS NOT NULL
```

The panel renders empty/errors. We don't have the customer's cluster or artifacts, so this is an analysis task to confirm the cause and decide whether a tool change is warranted.

**Distinct from #55** (`histogram_quantile()` → `PERCENTILE()`, closed and working). This is the *non-quantile* case: a panel referencing histogram **component series** (`_count`/`_sum`, and `_bucket` outside `histogram_quantile`) directly.

## What we found in the code so far

- The translator emits the metric name **unchanged** into the filter — there is no `_count`/`_sum` remap. `_resolve_metric_field()` returns the bare name for the OTel layout (`observability_migration/adapters/source/grafana/schema.py:574`), which lands in `| WHERE _count IS NOT NULL` (`.../translate.py:2514`).
- `_bucket` is stripped **only inside** `histogram_quantile` (`.../promql.py:1508`); there's no equivalent for bare `_count`/`_sum`.
- The rule pack treats `_count`/`_sum`/`_bucket` as counter suffixes for `rate()` purposes (`.../rules.py:37`), i.e. it **assumes those series exist as queryable fields** in the target.

## Open questions for engineering

- How does the OTel/prometheusreceiver ingestion path actually represent a Prometheus histogram in the target index? (Working theory: the receiver reassembles `_bucket`/`_sum`/`_count` into a single OTLP histogram → one `histogram`/`aggregate_metric_double` field named ``, with **no** `_count`/`_sum` scalar fields — so the emitted filter can never match. To be confirmed against a real index mapping.)
- If confirmed: is this a **translator/field-mapping gap** (remap `_count`→count/`value_count`, `_sum`→sum sub-field) or better handled elsewhere? What about derived panels — average (`rate(_sum)/rate(_count)`) and throughput (`rate(_count)`)?
- If instead the metric is simply not ingested in that environment, this is a **data-readiness gap**, not a tool bug — decide accordingly.
- Either way: should an unresolvable component-series reference **degrade with an honest warning** instead of silently emitting a filter on a nonexistent field?

## Who can help

**Jonas Kunz** can help us understand how histograms are represented (OTel histogram encoding + how the prometheusreceiver/ES exporter store `_count`/`_sum`/`_bucket`) — worth pulling him in to confirm the target field layout before deciding on a tool change.

## Suggested verification

Reproduce locally by seeding a histogram through the OTel/Prometheus benchmark pipeline; inspect the resulting index mapping and run the render audit to confirm whether the panel populates.

## Related

- #270 — sibling gap on the same `metrics.`-nested OTel indices (bare metric name needs the `metrics.` prefix).
- #262 (merged) — existing "warn on unverified OTel field default" path; the "degrade with an honest warning" option above should reuse it.

---

🔒 **Internal context (Elastic only):** originating discussion in a [private Slack thread 🔒 Elastic only](https://elastic.slack.com/archives/C075J1DPFAL/p1783073934624499?thread_ts=1781245716.965529&cid=C075J1DPFAL)

Contributor guide

Open the contributing guide

Research direction

Start with _resolve_metric_field() in observability_migration/adapters/source/grafana/schema.py:574, the filter generation in translate.py:2514, histogram handling in promql.py:1508, and suffix rules in rules.py:37. Seed a histogram through the OTel/Prometheus benchmark pipeline, inspect the resulting index mapping, and run the render audit. Done means confirming the target representation and deciding whether this is a translator gap, data-readiness issue, or warning path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.