elastic / elastic/observability-migration-platform
ES|QL time buckets should match Grafana auto vs fixed interval
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 23
Description
## Summary
After migrating a Grafana timeseries panel, the Kibana query is **ES|QL TS** with a hardcoded calendar bucket:
```esql
… BY time_bucket = TBUCKET(5 minute), device
```
In Grafana the panel did **not** hardcode a time bucket — resolution follows the dashboard time range. `TBUCKET(5 minute)` stays fixed when you zoom, so it does not behave like Grafana auto.
### Why ES|QL and not native PROMQL?
Source PromQL is:
`node_disk_read_bytes_total{device="$device_filtered"}`
The label matcher uses a Grafana template variable (`$device_filtered`). The native PROMQL path **intentionally skips** those panels so Kibana dashboard controls can bind via a visible ES|QL `WHERE … ?device_filtered` clause (params inside an opaque `PROMQL …` command do not bind — issue #230). Even with `--translation-mode native`, this panel falls through to ES|QL.
### ES|QL still needs a time bucket for XY charts
For a timeseries panel the TS ES|QL path must group by time. Mirror the same Grafana → resolution rule as native PROMQL (#318):
| Grafana source | ES\|QL should emit | Behavior |
| --- | --- | --- |
| **Auto** (no panel `interval`) | `TBUCKET(100, ?_tstart, ?_tend)` | ~100 buckets across the dashboard range — adapts when you zoom |
| Explicit panel `interval` (e.g. `1h`, `5m`) | `TBUCKET()` e.g. `TBUCKET(1 hour)`, `TBUCKET(5 minute)` | Fixed bucket **size** matching Grafana’s step |
| Today (always) | `TBUCKET(5 minute)` | Hardcoded — wrong for auto **and** wrong when Grafana set a different interval |
Verified on ES 9.5: `TBUCKET(100, ?_tstart, ?_tend)` works; bare `TBUCKET(100)` is rejected (count form needs the range args).
## Reproduction
```bash
obs-migrate migrate \
--source grafana \
--assets dashboards \
--input-mode files \
--input-dir /path/to/folder/with/dashboard.json \
--kibana-url "$KIBANA_URL" \
--kibana-api-key "$KIBANA_API_KEY" \
--es-url "$ES_URL" \
--es-api-key "$ES_API_KEY" \
--upload
```
Source Grafana dashboard JSON
```json
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 11,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "node_disk_read_bytes_total{device=\"$device_filtered\"}",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "__auto",
"range": true,
"refId": "A",
"useBackend": false
}
],
"title": "Panel Title",
"type": "timeseries"
}
],
"preload": false,
"schemaVersion": 41,
"tags": [],
"templating": {
"list": [
{
"current": {
"text": [
"/dev"
],
"value": [
"/dev"
]
},
"definition": "label_values(device)",
"multi": true,
"name": "device_all",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(device)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
},
{
"current": {
"text": [
"vda"
],
"value": [
"vda"
]
},
"definition": "label_values(node_disk_read_bytes_total{device!=\"nbd1\"},device)",
"multi": true,
"name": "device_filtered",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(node_disk_read_bytes_total{device!=\"nbd1\"},device)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
}
]
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "Issue 1",
"uid": "afst21a0rak8wf",
"version": 6
}
```
### What we see after migration
Migrated **ES|QL** (not PROMQL) query hardcodes `TBUCKET(5 minute)`:
```esql
TS metrics-*
| WHERE device == ?device_filtered
| WHERE node_disk_read_bytes_total IS NOT NULL
| STATS node_disk_read_bytes_total = MAX(LAST_OVER_TIME(node_disk_read_bytes_total)) BY time_bucket = TBUCKET(5 minute), device
| SORT time_bucket ASC
```
Source Grafana query had no hardcoded interval:
`node_disk_read_bytes_total{device="$device_filtered"}`
## Acceptance criteria
1. **Explain / keep the ES|QL fallthrough** when the PromQL label matcher has a template variable (control binding). Do not silently look like a PROMQL miss — the report should say why native PROMQL was skipped (see also #319 / #230).
2. **ES|QL XY bucket resolution must follow Grafana:**
- **Auto** (no panel `interval`): emit `TBUCKET(100, ?_tstart, ?_tend)` — not a hardcoded duration.
- **Explicit interval** (panel `interval` / query step, e.g. `1h`, `30m`, `5m`): emit `TBUCKET` with that **bucket size** (e.g. `TBUCKET(1 hour)`), not `5 minute` and not the count form.
- Do not always emit `TBUCKET(5 minute)`.
3. **Check in Kibana**
- Auto panel: changing the dashboard time range changes effective resolution (more/fewer points).
- Fixed-interval panel: bucket width stays the Grafana interval when zooming.
Same policy as #318 (`step=` only when Grafana set interval; bare/adaptive otherwise), applied to the ES|QL `TBUCKET` path.
Contributor guide
Research direction
Start from the Grafana-to-ES|QL migration path exercised by the provided obs-migrate command, then trace how timeseries grouping emits TBUCKET and how panels expose interval settings. Check the existing native PROMQL fallthrough and its reporting for template-variable matchers. Done means auto panels emit TBUCKET(100, ?_tstart, ?_tend), explicit intervals preserve their bucket size, and both cases are verified in Kibana.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100