elastic / elastic/observability-migration-platform
Investigate: histogram component-series (_count/_sum) references produce empty panels on OTel targets
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 23
Description
## Context
Surfaced in a customer Grafana→Kibana migration. The histogram `kubelet_pod_start_duration_seconds` produced a panel whose migrated ES|QL is:
```
FROM metrics-prometheusreceiver.otel-*
| WHERE kubelet_active_pods IS NOT NULL AND kubelet_pod_start_duration_seconds_count IS NOT NULL
```
The panel renders empty/errors. We don't have the customer's cluster or artifacts, so this is an analysis task to confirm the cause and decide whether a tool change is warranted.
**Distinct from #55** (`histogram_quantile()` → `PERCENTILE()`, closed and working). This is the *non-quantile* case: a panel referencing histogram **component series** (`_count`/`_sum`, and `_bucket` outside `histogram_quantile`) directly.
## What we found in the code so far
- The translator emits the metric name **unchanged** into the filter — there is no `_count`/`_sum` remap. `_resolve_metric_field()` returns the bare name for the OTel layout (`observability_migration/adapters/source/grafana/schema.py:574`), which lands in `| WHERE _count IS NOT NULL` (`.../translate.py:2514`).
- `_bucket` is stripped **only inside** `histogram_quantile` (`.../promql.py:1508`); there's no equivalent for bare `_count`/`_sum`.
- The rule pack treats `_count`/`_sum`/`_bucket` as counter suffixes for `rate()` purposes (`.../rules.py:37`), i.e. it **assumes those series exist as queryable fields** in the target.
## Open questions for engineering
- How does the OTel/prometheusreceiver ingestion path actually represent a Prometheus histogram in the target index? (Working theory: the receiver reassembles `_bucket`/`_sum`/`_count` into a single OTLP histogram → one `histogram`/`aggregate_metric_double` field named ``, with **no** `_count`/`_sum` scalar fields — so the emitted filter can never match. To be confirmed against a real index mapping.)
- If confirmed: is this a **translator/field-mapping gap** (remap `_count`→count/`value_count`, `_sum`→sum sub-field) or better handled elsewhere? What about derived panels — average (`rate(_sum)/rate(_count)`) and throughput (`rate(_count)`)?
- If instead the metric is simply not ingested in that environment, this is a **data-readiness gap**, not a tool bug — decide accordingly.
- Either way: should an unresolvable component-series reference **degrade with an honest warning** instead of silently emitting a filter on a nonexistent field?
## Who can help
**Jonas Kunz** can help us understand how histograms are represented (OTel histogram encoding + how the prometheusreceiver/ES exporter store `_count`/`_sum`/`_bucket`) — worth pulling him in to confirm the target field layout before deciding on a tool change.
## Suggested verification
Reproduce locally by seeding a histogram through the OTel/Prometheus benchmark pipeline; inspect the resulting index mapping and run the render audit to confirm whether the panel populates.
## Related
- #270 — sibling gap on the same `metrics.`-nested OTel indices (bare metric name needs the `metrics.` prefix).
- #262 (merged) — existing "warn on unverified OTel field default" path; the "degrade with an honest warning" option above should reuse it.
---
🔒 **Internal context (Elastic only):** originating discussion in a [private Slack thread 🔒 Elastic only](https://elastic.slack.com/archives/C075J1DPFAL/p1783073934624499?thread_ts=1781245716.965529&cid=C075J1DPFAL)
Contributor guide
Research direction
Start with _resolve_metric_field() in observability_migration/adapters/source/grafana/schema.py:574, the filter generation in translate.py:2514, histogram handling in promql.py:1508, and suffix rules in rules.py:37. Seed a histogram through the OTel/Prometheus benchmark pipeline, inspect the resulting index mapping, and run the render audit. Done means confirming the target representation and deciding whether this is a translator gap, data-readiness issue, or warning path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100