elastic / elastic/observability-migration-platform

Grafana variable→Kibana control fidelity: inconsistent control emission + dropped inter-variable label filter

Open
#269 1 comment 0 reactions 0 assignees View on GitHub
asset:dashboards program:grafana-engine quality source:grafana workstream:translator
Dominant language
Python
Stars
6
Forks
8
Avg merge
2d 22h
Merged PRs (30d)
23

Description

## Problem

Migrating Grafana **query template variables** that are metric-scoped and label-filtered produces Kibana Controls that don't preserve the source variable behavior. Surfaced from a customer Grafana→Kibana review (label-based panel filtering). Two distinct defects, both reproduced end-to-end on a live target.

## Reproduction

Source dashboard has two chained variables and one panel:

- `$instance` = `label_values(container_memory_cache, instance)`
- `$id` = `label_values(container_memory_cache{instance="$instance"}, id)` ← metric-scoped **and** filtered by `$instance`
- panel = `avg(container_memory_cache{id="$id"})`

Reproduction Grafana dashboard JSON

```json
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "Repro for migration gap: query-based template variable (metric-scoped, label-filtered) has no Kibana Controls equivalent.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"links": [],
"panels": [
{
"datasource": {
"uid": "efpaoxhs2fshsf"
},
"description": "Container memory cache usage in bytes\n\n**Type:** *gauge*\n\n",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 9,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
},
"unit": "bytes"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 2,
"maxDataPoints": 500,
"options": {
"annotations": {
"multiLane": true
},
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.3.3",
"targets": [
{
"editorMode": "builder",
"expr": "avg(container_memory_cache{id=\"$id\"})",
"fromExploreMetrics": false,
"legendFormat": "avg",
"range": true,
"refId": "container_memory_cache-avg"
}
],
"title": "container_memory_cache",
"type": "timeseries"
}
],
"preload": false,
"refresh": "1m",
"schemaVersion": 42,
"tags": [
"repro",
"label-filter",
"obs-migration"
],
"templating": {
"list": [
{
"current": {
"text": "cadvisor:8080",
"value": "cadvisor:8080"
},
"definition": "label_values(container_memory_cache,instance)",
"name": "instance",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(container_memory_cache,instance)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
},
{
"current": {
"text": "id_1",
"value": "id_1"
},
"definition": "label_values(container_memory_cache{instance=\"$instance\"},id)",
"name": "id",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(container_memory_cache{instance=\"$instance\"},id)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
}
]
},
"time": {
"from": "now-15m",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Label-filter control repro (container_memory_cache)",
"uid": "label-filter-repro-01",
"weekStart": ""
}
```

```
# offline
obs-migrate migrate --source grafana --input-mode files --input-dir --assets dashboards
# complete (target schema discovery)
obs-migrate migrate --source grafana --input-mode files --input-dir --assets dashboards \
--es-url --es-api-key --upload --kibana-url --kibana-api-key
```

## Defect 1 — control emission depends on target schema, inconsistently

- **Offline run:** emits **2** controls (`instance`, `id`).
- **Complete run (`--es-url`):** emits **1** control (`id`); the `instance` control is dropped because schema-discovery found `service.instance.id` absent in the target.

**Impact:** the same source dashboard yields structurally different dashboards depending on whether a target was reachable, and a control silently disappears with no warning surfaced on the panel/control. A field that is merely not yet ingested (data-readiness) causes a permanent control drop at migrate time.

## Defect 2 — inter-variable (cascading) label filter is not translated

Grafana `$id` options are scoped to the selected `$instance`. The migrated `id` control query drops that scope:

```esql
# emitted
FROM metrics-* | WHERE container_memory_cache IS NOT NULL AND id IS NOT NULL
| STATS count = COUNT(*) BY id | SORT id ASC | KEEP id | LIMIT 1000
# expected: options constrained by the selected instance, e.g. an added
# WHERE service.instance.id == ?instance (or equivalent control dependency)
```

**Impact:** the `id` control lists every value regardless of the selected instance, and the `instance` control affects nothing — it neither filters the panel (correct: `$instance` was never a panel filter in the source) nor narrows `$id` (the actual regression). The control is present but inert.

## Expected

- Control emission should be deterministic and should not silently drop a source variable purely because the target field is not yet populated; degradation should be surfaced (warning) rather than a vanished control.
- Cascading variable relationships (`$instance` scoping `$id`) should be translated into dependent Kibana controls or explicitly reported as an unsupported gap.

## Acceptance criteria

- [ ] Documented current vs. desired behavior for metric-scoped, label-filtered, and **chained** query variables
- [ ] Decision on whether target-schema absence should drop vs. warn-and-keep a control (offline/complete consistency)
- [ ] Recommendation for representing inter-variable dependencies in Kibana Controls (dependent controls, ES|QL param, or documented degradation)
- [ ] Reproduction fixture + before/after control ES|QL captured

## Related

- #270, #271 — sibling OpenTelemetry `metrics.`-prefixed field-resolution gaps from the same migration review.
- #262 (merged) — added the "warn on unverified OTel field default" path; the warn-and-keep-a-control degradation asked for above should build on that mechanism rather than add a parallel one.

---

🔒 **Internal context (Elastic only):** originating discussion in a [private Slack thread 🔒 Elastic only](https://elastic.slack.com/archives/C075J1DPFAL/p1783073934624499?thread_ts=1781245716.965529&cid=C075J1DPFAL)

Contributor guide

Open the contributing guide

Research direction

Reproduce the offline and complete migration commands against the supplied Grafana fixture, then trace control emission, target-schema discovery, and query-variable handling, including the warning path added in #262. Done means documenting current and desired behavior, deciding drop versus warn-and-keep, specifying dependency handling, and capturing the fixture with before/after control ES|QL.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
observability-sre, tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.