elastic / elastic/observability-migration-platform
Grafana: topk()/bottomk() migrates as snapshot, not time-series (temporal loss)
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 23
Description
#### Problem statement
Grafana's `topk(N, expr)` and `bottomk(N, expr)` functions are used to display the top or bottom N time-series lines over a selected time window — for example, "the 5 busiest pods by CPU over the last hour" shown as 5 continuous trend lines. The migration tool currently translates these expressions to a valid ES|QL query, but the result is semantically different: instead of N time-series lines, the Kibana panel shows N individual data points representing the most recent value per series, sorted by that value. The temporal history — the defining characteristic of these panels — is lost.
---
#### Known cases
The ES|QL translation path is used for all `topk()`/`bottomk()` expressions (native PromQL blocks them as unsupported constructs).
| Expression | Native PromQL path | ES\|QL path (current) | ES\|QL path (after fix) |
|---|---|---|---|
| `topk(5, rate(process_cpu_seconds_total[5m]))` | Blocked — `topk` unsupported | ⚠️ migrated with warnings — single snapshot, no time-series | ⚠️ migrated with warnings — N series by most recent value; time-series history not replicated |
| `bottomk(5, rate(process_cpu_seconds_total[5m]))` | Blocked — `bottomk` unsupported | ⚠️ migrated with warnings — single snapshot, no time-series | ⚠️ migrated with warnings — N series by most recent value; time-series history not replicated |
Example dashboards:
- [Calico: Felix](https://grafana.com/grafana/dashboards/12175) — *Felix restarts/hr (top N)*, *CPU Usage (top N)*, *Open file descriptors (top N)*
- [VictoriaMetrics - vmalert](https://grafana.com/grafana/dashboards/14950) — *Top groups avg evaluation duration*, *Top active alerts*
Note: `topk($variable_limit, expr)` (template variable used as the limit argument) is a separate parse-failure bug tracked in [#58](https://github.com/elastic/observability-migration-platform/issues/58).
---
#### Verification
**Check 1 — Routing**
```
topk(5, rate(process_cpu_seconds_total[5m])):
can_use_native_promql = False
_native_promql_has_distinct_metric_arithmetic = False
→ routing: esql_translation
```
**Check 2 — Current CLI behaviour**
```
feasibility = feasible
warnings = ['topk() without group labels: collapsed to single-series top N;
add preferred_group_labels hint for per-series breakdown']
esql_query:
TS metrics-*
| WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend
| WHERE process_cpu_seconds_total IS NOT NULL
| STATS _bucket_value = AVG(RATE(process_cpu_seconds_total, 5m))
BY time_bucket = TBUCKET(5 minute)
| SORT time_bucket ASC
| STATS value = LAST(_bucket_value, time_bucket)
| SORT value DESC
| LIMIT 5
```
The generated query takes the last bucket value per series and returns the top 5 by most-recent value. A single aggregate point is returned per series rather than the full time-series history.
**Check 3 — ES|QL function check**
```
ROW val = 2.0 | EVAL r = TOPK(val, 5)
→ HTTP 400: Unknown function [TOPK], did you mean [top]?
```
ES|QL does not have a `TOPK` scalar function. The current CLI correctly avoids it and uses `SORT … LIMIT` instead. Producing the correct time-series top-N result requires a two-pass query: first identify the top N label sets across the full time range, then fetch their complete time buckets.
**Check 4 — Grafana baseline**
```
topk(5, rate(process_cpu_seconds_total[5m])):
STATUS: success | 1 series | value: [1779905664, '0.0086']
Labels: {instance: "localhost:9090", job: "prometheus"}
bottomk(5, rate(process_cpu_seconds_total[5m])):
STATUS: success | 1 series | value: [1779905664, '0.0086']
```
Both expressions return valid data. With only 1 series available in the local Prometheus, topk and bottomk both return that same series — ranking becomes meaningful with multiple series in production.
---
#### Expected behaviour
ES|QL does not support a true time-series top-N query (no equivalent to PromQL's per-step ranking). After the fix, the panel remains `migrated_with_warnings` but with an honest warning that clearly describes what the output actually is: N series ranked by their most recent value, not continuous trend lines. Users can then decide whether this approximation is acceptable. The current warning (`'topk() without group labels: collapsed to single-series top N'`) is replaced with an explicit message such as: `'topk()/bottomk() shows N series ordered by most recent value only — time-series history from Grafana is not replicated'`.
Contributor guide
Assessment
This issue has not been assessed yet.