elastic / elastic/observability-migration-platform

Native-PROMQL panels render empty on first open: migrated controls carry no default selection (Kibana 9.5 does bind PROMQL params)

Open
#373 0 comments 0 reactions 0 assignees View on GitHub
asset:dashboards bug program:grafana-engine source:grafana
Dominant language
Python
Stars
6
Forks
8
Avg merge
2d 22h
Merged PRs (30d)
23

Description

## Summary

Native-PROMQL panels bind their Grafana template variables as `?params` inside the PROMQL expression. Kibana 9.5 **does** forward dashboard control values into those params (see verification below, which answers the open question in #319 / #345). But the migrated control is emitted with **no default selection**, so on first open the param binds to the empty string, `label=~""` matches no series, and every native-PROMQL panel renders "No results found".

Selecting any value in the control fixes it immediately. The dashboard is therefore fully functional but looks broken on arrival — which is exactly the impression reported as *"dashboards generated with the PromQL option don't work well"* in #131.

The ES|QL path does not hit this because its emitted predicate tolerates the empty binding: `(?Node == "" OR (node RLIKE ?Node OR ...))`. Native PROMQL has no such empty-tolerance.

## Verification that Kibana 9.5 binds PROMQL params (re: #319, #345)

#319 asks to re-verify whether a Lens dashboard panel with native PROMQL plus a matching control actually binds on current Kibana/ES. On Elasticsearch 9.5.0-SNAPSHOT + Kibana 9.5.0: **yes, it binds.**

Evidence — same panel, same dashboard, only the control selection changed:

| Control state | Panel result |
|---|---|
| `K8S` control unselected (as uploaded) | "No results found" |
| `K8S` control set to `k8s-prod` | renders correctly, 4 series |

Panels confirmed to come alive on selection: **Namespaces CPU Usage kernel(>0.5)**, **Namespaces WSS Memory Usage (>1G)**.

The same query run directly against `/_query` with the param bound returns 108 rows, and returns identical results whether bound to `.*`, to `k8s-prod`, or with the matcher removed entirely — so the query itself was never the problem.

So the `_promql_label_matcher_has_template_variable` guard discussed in #319/#345 looks stale for 9.5; the remaining gap is the **missing default selection**, not param forwarding.

## Impact

On [dashboard 15661](https://grafana.com/grafana/dashboards/15661-k8s-dashboard-en-20250125/), all **8** native-PROMQL panels reference `?origin_prometheus` and all 8 are empty on first open. That is 100% of the native-PROMQL panels — i.e. exactly the panels the migration summary describes as the *numerically verifiable* ones:

> Native-PROMQL panels are **numerically verifiable** via the native-PROMQL oracle. ES|QL-translated panels are structural-only unless separately validated.

Correlation across the dashboard is exact — every empty panel is native-PROMQL with `?origin_prometheus`; every rendering panel is ES|QL without it.

## Secondary defect: the run reports a control that was in fact emitted

The migration prints:

```
CONTROL WARNINGS (1):
[K8S Dashboard] variable 'origin_prometheus' could not resolve source field 'origin_prometheus'
to a supported target field; no Kibana control was emitted
```

But the uploaded dashboard **does** contain that control (labelled `K8S`), populated with 2 options (`.*`, `k8s-prod`). It is synthesized later by `_ensure_param_controls` — a contradiction the code already anticipates in a comment at `panels.py` ~8404. The warning misleads an operator into thinking the dropdown is missing rather than merely unset.

The underlying reason `resolve_control_field` returned `None` is that `origin_prometheus` is hard-coded into the default `ignored_labels`:

```python
# observability_migration/adapters/source/grafana/rules.py ~153
ignored_labels: list = field(default_factory=lambda: [
"origin_prometheus",
# Prometheus scrape-target metadata labels — these describe the scrape
# configuration (endpoint path, scheme) and are not stored as metric
# dimensions in Elastic indices. ...
"metrics_path",
"__metrics_path__",
])
```

Unlike its neighbours it carries no rationale, and the neighbours' rationale does not apply to it: on this target `origin_prometheus` **is** a real, aggregatable `keyword` dimension (`_field_caps` confirms both `origin_prometheus` and `labels.origin_prometheus`, `aggregatable: true`, `time_series_dimension: true`).

Worth noting for a multi-cluster federation target: silently dropping `origin_prometheus=~"$origin_prometheus"` from a query would merge data across Prometheus origins and inflate every aggregate. In this single-origin test that is a no-op, but it looks like a correctness risk in the setups where the label exists.

## Suggested fix

1. Emit a concrete default selection for every control a native-PROMQL panel binds, mirroring the source variable's default (`current`, else `All` → `.*`, else first option). This is item 2 of #131, which was closed.
2. Reconsider the unconditional `origin_prometheus` entry in `ignored_labels` — at minimum, don't ignore a label the live target advertises as an aggregatable dimension.
3. Make the control warning reflect the final emitted dashboard rather than the first resolution attempt.

## Environment

- Engine: `main`
- Elasticsearch 9.5.0-SNAPSHOT / Kibana 9.5.0 (local `elastic-package` stack)
- Target layout: native `/_prometheus` remote-write data stream (`metrics-k8s.prometheus-parity`)

Related: #131 (closed), #319, #345

Contributor guide

Open the contributing guide

Research direction

Start with observability_migration/adapters/source/grafana/rules.py and panels.py around _ensure_param_controls and the noted ~8404 comment; trace how origin_prometheus is resolved, defaulted, and reported. Done means native-PROMQL panels have a valid initial control selection, warnings match the emitted dashboard, and the ignored-label behavior is verified against the described aggregatable field.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, grafana, prometheus, python
Domain
observability-sre, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.