elastic / elastic/observability-migration-platform

ES|QL time buckets should match Grafana auto vs fixed interval

Open
#316 1 comment 0 reactions 0 assignees View on GitHub
asset:dashboards bug phase:3 program:grafana-engine quality source:grafana workstream:translator
Dominant language
Python
Stars
6
Forks
8
Avg merge
2d 22h
Merged PRs (30d)
23

Description

## Summary

After migrating a Grafana timeseries panel, the Kibana query is **ES|QL TS** with a hardcoded calendar bucket:

```esql
… BY time_bucket = TBUCKET(5 minute), device
```

In Grafana the panel did **not** hardcode a time bucket — resolution follows the dashboard time range. `TBUCKET(5 minute)` stays fixed when you zoom, so it does not behave like Grafana auto.

### Why ES|QL and not native PROMQL?

Source PromQL is:

`node_disk_read_bytes_total{device="$device_filtered"}`

The label matcher uses a Grafana template variable (`$device_filtered`). The native PROMQL path **intentionally skips** those panels so Kibana dashboard controls can bind via a visible ES|QL `WHERE … ?device_filtered` clause (params inside an opaque `PROMQL …` command do not bind — issue #230). Even with `--translation-mode native`, this panel falls through to ES|QL.

### ES|QL still needs a time bucket for XY charts

For a timeseries panel the TS ES|QL path must group by time. Mirror the same Grafana → resolution rule as native PROMQL (#318):

| Grafana source | ES\|QL should emit | Behavior |
| --- | --- | --- |
| **Auto** (no panel `interval`) | `TBUCKET(100, ?_tstart, ?_tend)` | ~100 buckets across the dashboard range — adapts when you zoom |
| Explicit panel `interval` (e.g. `1h`, `5m`) | `TBUCKET()` e.g. `TBUCKET(1 hour)`, `TBUCKET(5 minute)` | Fixed bucket **size** matching Grafana’s step |
| Today (always) | `TBUCKET(5 minute)` | Hardcoded — wrong for auto **and** wrong when Grafana set a different interval |

Verified on ES 9.5: `TBUCKET(100, ?_tstart, ?_tend)` works; bare `TBUCKET(100)` is rejected (count form needs the range args).

## Reproduction

```bash
obs-migrate migrate \
--source grafana \
--assets dashboards \
--input-mode files \
--input-dir /path/to/folder/with/dashboard.json \
--kibana-url "$KIBANA_URL" \
--kibana-api-key "$KIBANA_API_KEY" \
--es-url "$ES_URL" \
--es-api-key "$ES_API_KEY" \
--upload
```

Source Grafana dashboard JSON

```json
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 11,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "node_disk_read_bytes_total{device=\"$device_filtered\"}",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "__auto",
"range": true,
"refId": "A",
"useBackend": false
}
],
"title": "Panel Title",
"type": "timeseries"
}
],
"preload": false,
"schemaVersion": 41,
"tags": [],
"templating": {
"list": [
{
"current": {
"text": [
"/dev"
],
"value": [
"/dev"
]
},
"definition": "label_values(device)",
"multi": true,
"name": "device_all",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(device)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
},
{
"current": {
"text": [
"vda"
],
"value": [
"vda"
]
},
"definition": "label_values(node_disk_read_bytes_total{device!=\"nbd1\"},device)",
"multi": true,
"name": "device_filtered",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(node_disk_read_bytes_total{device!=\"nbd1\"},device)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
}
]
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "Issue 1",
"uid": "afst21a0rak8wf",
"version": 6
}
```

### What we see after migration

Migrated **ES|QL** (not PROMQL) query hardcodes `TBUCKET(5 minute)`:

```esql
TS metrics-*
| WHERE device == ?device_filtered
| WHERE node_disk_read_bytes_total IS NOT NULL
| STATS node_disk_read_bytes_total = MAX(LAST_OVER_TIME(node_disk_read_bytes_total)) BY time_bucket = TBUCKET(5 minute), device
| SORT time_bucket ASC
```

Source Grafana query had no hardcoded interval:

`node_disk_read_bytes_total{device="$device_filtered"}`

## Acceptance criteria

1. **Explain / keep the ES|QL fallthrough** when the PromQL label matcher has a template variable (control binding). Do not silently look like a PROMQL miss — the report should say why native PROMQL was skipped (see also #319 / #230).

2. **ES|QL XY bucket resolution must follow Grafana:**
- **Auto** (no panel `interval`): emit `TBUCKET(100, ?_tstart, ?_tend)` — not a hardcoded duration.
- **Explicit interval** (panel `interval` / query step, e.g. `1h`, `30m`, `5m`): emit `TBUCKET` with that **bucket size** (e.g. `TBUCKET(1 hour)`), not `5 minute` and not the count form.
- Do not always emit `TBUCKET(5 minute)`.

3. **Check in Kibana**
- Auto panel: changing the dashboard time range changes effective resolution (more/fewer points).
- Fixed-interval panel: bucket width stays the Grafana interval when zooming.

Same policy as #318 (`step=` only when Grafana set interval; bare/adaptive otherwise), applied to the ES|QL `TBUCKET` path.

Contributor guide

Open the contributing guide

Research direction

Start from the Grafana-to-ES|QL migration path exercised by the provided obs-migrate command, then trace how timeseries grouping emits TBUCKET and how panels expose interval settings. Check the existing native PROMQL fallthrough and its reporting for template-variable matchers. Done means auto panels emit TBUCKET(100, ?_tstart, ?_tend), explicit intervals preserve their bucket size, and both cases are verified in Kibana.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.