elastic / elastic/observability-migration-platform
Grafana: anchored template variable `label=~"^$Var$"` migrates to the literal `^1$`, silently emptying panels
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 23
Description
## Summary
A Grafana template variable used inside an **anchored** PromQL regex — `label=~"^$Var$"` — is replaced with the literal string `1` instead of being bound to a control parameter. The emitted query becomes `label=~"^1$"`, which matches nothing, so the panel renders empty.
The panel is still reported as **`migrated` / Green** with no warning, so nothing in the run output signals the loss.
`^$Var$` is the idiomatic Grafana exact-match form and is very common in public dashboards.
## Reproduction
Minimal one-panel dashboard, variable `Node` (`type: query`, `includeAll: true`, `allValue: ".*"`):
```bash
obs-migrate migrate --source grafana --input-mode files \
--input-dir ./input --output-dir ./out --assets dashboards \
--data-view metrics-k8s.prometheus-parity --esql-index metrics-k8s.prometheus-parity
```
| Source PromQL | Emitted |
|---|---|
| `sum(kube_node_status_allocatable{resource="cpu", node=~"$Node"}) by (node)` | `... node=~?Node ...` — correct, bound param |
| `sum(kube_node_status_allocatable{resource="cpu", node=~"^$Node$"}) by (node)` | `... node=~"^1$" ...` — **variable replaced with literal `1`** |
The substitution is a constant: it is still `^1$` when the variable has a real `current` value (`{"text":"k8s-node-1","value":"k8s-node-1"}`) and regardless of `refresh`.
## Impact — measured
Against a live ES 9.5.0 target with real data, the emitted vs. corrected selector:
```
AS EMITTED node=~"^1$" rows=0 nodes=[]
CORRECTED node=~"^.*$" rows=108 nodes=['k8s-master-1','k8s-node-1','k8s-node-2']
```
On [dashboard 15661](https://grafana.com/grafana/dashboards/15661-k8s-dashboard-en-20250125/) (K8S Dashboard, ~2.9M downloads) this silently empties **4 panels**, all graded `migrated` with a Green verification gate and no warning:
- Nodes CPU Breakdown
- Node Memory Breakdown
- Pod Containers CPU Utilization (Maximum100%Associable Nodes)
- Pod Containers RSS Memory Usage (Associable Nodes)
## Root cause
`observability_migration/adapters/source/grafana/panels.py`
The label-matcher handling only fires when the matcher value is *exactly* a variable, because `grafana_template_var_name()` / `_GRAFANA_VAR_TOKEN_PATTERN` are anchored to the whole value:
```python
# ~1733
if is_feature_supported(runtime_features, PROMQL_LABEL_MATCHER_PARAMS):
expr = _promql_label_matcher_vars_to_params(expr, regex_default_params)
# ~1737 — only matches a value that is *only* the variable
expr = re.sub(rf'=~"\s*{_GRAFANA_VAR_TOKEN_PATTERN}\s*"', '=~".*"', expr)
```
`^$Node$` fails both, falls through, and is then caught by the scalar-arithmetic fallback:
```python
# ~1751
# Remove any remaining bare $variable tokens (e.g. in arithmetic).
# Multiplicative identity preserves magnitude better than 0.
expr = _GRAFANA_VAR_PLAIN_RE.sub("1", expr)
```
`1` is a sensible identity for `* $factor` in arithmetic, but inside a label-matcher value it silently produces a filter that matches the literal string `1`.
## Suggested fix
The anchor-stripping helpers already exist for the ES|QL path — `_strip_promql_regex_anchors()` and `_unanchored_template_var_name()` (`promql.py` ~993, `panels.py` ~9698). Apply the same normalization on the native-PROMQL path so `^$Var$` is treated like `$Var` for a `=~`/`!~` matcher (binding `?Var`, or falling back to `.*`).
Failing that, the scalar `$var → 1` fallback should at minimum not apply inside a label-matcher value, and hitting it should raise a warning rather than pass as Green.
## Environment
- Engine: `main`
- Elasticsearch 9.5.0-SNAPSHOT / Kibana 9.5.0 (local `elastic-package` stack)
- Target layout: native `/_prometheus` remote-write data stream (`metrics-k8s.prometheus-parity`)
Contributor guide
Research direction
Start in observability_migration/adapters/source/grafana/panels.py around the native-PromQL handling near lines 1733-1751, then read _strip_promql_regex_anchors() and _unanchored_template_var_name() in promql.py and panels.py. Use the supplied obs-migrate reproduction with an anchored ^$Node$ matcher; done means it binds the variable or safely falls back to .* instead of emitting ^1$, with the panel no longer silently empty.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana, prometheus, python
- Domain
- observability-sre, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100