elastic / elastic/observability-migration-platform
Native-PROMQL panels render empty on first open: migrated controls carry no default selection (Kibana 9.5 does bind PROMQL params)
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 23
Description
## Summary
Native-PROMQL panels bind their Grafana template variables as `?params` inside the PROMQL expression. Kibana 9.5 **does** forward dashboard control values into those params (see verification below, which answers the open question in #319 / #345). But the migrated control is emitted with **no default selection**, so on first open the param binds to the empty string, `label=~""` matches no series, and every native-PROMQL panel renders "No results found".
Selecting any value in the control fixes it immediately. The dashboard is therefore fully functional but looks broken on arrival — which is exactly the impression reported as *"dashboards generated with the PromQL option don't work well"* in #131.
The ES|QL path does not hit this because its emitted predicate tolerates the empty binding: `(?Node == "" OR (node RLIKE ?Node OR ...))`. Native PROMQL has no such empty-tolerance.
## Verification that Kibana 9.5 binds PROMQL params (re: #319, #345)
#319 asks to re-verify whether a Lens dashboard panel with native PROMQL plus a matching control actually binds on current Kibana/ES. On Elasticsearch 9.5.0-SNAPSHOT + Kibana 9.5.0: **yes, it binds.**
Evidence — same panel, same dashboard, only the control selection changed:
| Control state | Panel result |
|---|---|
| `K8S` control unselected (as uploaded) | "No results found" |
| `K8S` control set to `k8s-prod` | renders correctly, 4 series |
Panels confirmed to come alive on selection: **Namespaces CPU Usage kernel(>0.5)**, **Namespaces WSS Memory Usage (>1G)**.
The same query run directly against `/_query` with the param bound returns 108 rows, and returns identical results whether bound to `.*`, to `k8s-prod`, or with the matcher removed entirely — so the query itself was never the problem.
So the `_promql_label_matcher_has_template_variable` guard discussed in #319/#345 looks stale for 9.5; the remaining gap is the **missing default selection**, not param forwarding.
## Impact
On [dashboard 15661](https://grafana.com/grafana/dashboards/15661-k8s-dashboard-en-20250125/), all **8** native-PROMQL panels reference `?origin_prometheus` and all 8 are empty on first open. That is 100% of the native-PROMQL panels — i.e. exactly the panels the migration summary describes as the *numerically verifiable* ones:
> Native-PROMQL panels are **numerically verifiable** via the native-PROMQL oracle. ES|QL-translated panels are structural-only unless separately validated.
Correlation across the dashboard is exact — every empty panel is native-PROMQL with `?origin_prometheus`; every rendering panel is ES|QL without it.
## Secondary defect: the run reports a control that was in fact emitted
The migration prints:
```
CONTROL WARNINGS (1):
[K8S Dashboard] variable 'origin_prometheus' could not resolve source field 'origin_prometheus'
to a supported target field; no Kibana control was emitted
```
But the uploaded dashboard **does** contain that control (labelled `K8S`), populated with 2 options (`.*`, `k8s-prod`). It is synthesized later by `_ensure_param_controls` — a contradiction the code already anticipates in a comment at `panels.py` ~8404. The warning misleads an operator into thinking the dropdown is missing rather than merely unset.
The underlying reason `resolve_control_field` returned `None` is that `origin_prometheus` is hard-coded into the default `ignored_labels`:
```python
# observability_migration/adapters/source/grafana/rules.py ~153
ignored_labels: list = field(default_factory=lambda: [
"origin_prometheus",
# Prometheus scrape-target metadata labels — these describe the scrape
# configuration (endpoint path, scheme) and are not stored as metric
# dimensions in Elastic indices. ...
"metrics_path",
"__metrics_path__",
])
```
Unlike its neighbours it carries no rationale, and the neighbours' rationale does not apply to it: on this target `origin_prometheus` **is** a real, aggregatable `keyword` dimension (`_field_caps` confirms both `origin_prometheus` and `labels.origin_prometheus`, `aggregatable: true`, `time_series_dimension: true`).
Worth noting for a multi-cluster federation target: silently dropping `origin_prometheus=~"$origin_prometheus"` from a query would merge data across Prometheus origins and inflate every aggregate. In this single-origin test that is a no-op, but it looks like a correctness risk in the setups where the label exists.
## Suggested fix
1. Emit a concrete default selection for every control a native-PROMQL panel binds, mirroring the source variable's default (`current`, else `All` → `.*`, else first option). This is item 2 of #131, which was closed.
2. Reconsider the unconditional `origin_prometheus` entry in `ignored_labels` — at minimum, don't ignore a label the live target advertises as an aggregatable dimension.
3. Make the control warning reflect the final emitted dashboard rather than the first resolution attempt.
## Environment
- Engine: `main`
- Elasticsearch 9.5.0-SNAPSHOT / Kibana 9.5.0 (local `elastic-package` stack)
- Target layout: native `/_prometheus` remote-write data stream (`metrics-k8s.prometheus-parity`)
Related: #131 (closed), #319, #345
Contributor guide
Research direction
Start with observability_migration/adapters/source/grafana/rules.py and panels.py around _ensure_param_controls and the noted ~8404 comment; trace how origin_prometheus is resolved, defaulted, and reported. Done means native-PROMQL panels have a valid initial control selection, warnings match the emitted dashboard, and the ignored-label behavior is verified against the described aggregatable field.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, grafana, prometheus, python
- Domain
- observability-sre, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100