elastic / elastic/observability-migration-platform

ES|QL TS: counter column alongside CASE-filtered gauge columns in one STATS returns null gauges (CPU ratio panels show N/A)

Open
#372 0 comments 0 reactions 0 assignees View on GitHub
asset:dashboards bug program:grafana-engine source:grafana
Dominant language
Python
Stars
6
Forks
8
Avg merge
2d 22h
Merged PRs (30d)
23

Description

## Summary

When a multi-target panel is fused into a single `TS ... STATS`, and that `STATS` combines a **counter-typed** metric column with **gauge** columns whose value is wrapped in `CASE(, ...)`, the gauge columns come back `null`. The counter column is computed correctly.

The panel then renders `N/A` for every ratio that divides by a gauge. The migration reports it as migrated, and the query returns HTTP 200 — the nulls are silent.

## Observed

Live Kibana request for **Node CPU Number of cores** (captured from the dashboard):

```json
"values": [["Limit", null], ["Requests", null], ["Total Cores", null], ["Usage", 6.6800577646427755]]
```

`Usage` (the `IRATE` counter column) is correct and matches Grafana's `6.7`. `Total Cores` / `Requests` / `Limit` — all `LAST_OVER_TIME(CASE((resource == "cpu") and (unit == "core"), , NULL), 5m)` — are `null`, though the same gauges return `20.0 / 8.3 / 16.6` when queried on their own.

## Reproduction matrix

Same index, same time range, same data (`metrics-k8s.prometheus-parity`, ES 9.5.0-SNAPSHOT):

| # | Query shape | Result |
|---|---|---|
| A | CASE-wrapped gauges only (`limits`, `allocatable`) | `[16.6, 20.0]` ✅ |
| B | `SUM(IRATE(container_cpu_usage_seconds_total))` alone | `9.32` ✅ |
| C | **A + B in one `STATS`** (the emitted shape) | `[9.32, null, null]` ❌ |
| D | Same as C but all-gauge (memory equivalent) | `[1.16e10, 2.07e10, 4.29e10]` ✅ |
| E | `LAST_OVER_TIME()` + CASE-wrapped gauge | `[146969.98, null]` ❌ |
| F | `SUM(IRATE(counter))` + `SUM(LAST_OVER_TIME(gauge))` **without CASE** | `[15.87, 6.87e11]` ✅ |
| G | CASE-wrapped gauge only, `WHERE` widened to include the counter's documents | `[20.0]` ✅ |
| H | CASE-wrapped gauge only, narrow `WHERE` | `[20.0]` ✅ |

Reading of the matrix:

- **G vs H** rules out `WHERE` breadth / document-set size as the trigger.
- **D** rules out "multiple metrics in one `STATS`" — three gauges with CASE work fine.
- **F** rules out "counter and gauge in one `STATS`" on its own — without `CASE` both resolve.
- **C and E** share the one trigger: a **counter-typed** metric column present in the same `STATS` as a `CASE`-over-dimensions gauge column.

Case C, verbatim:

```esql
TS metrics-k8s.prometheus-parity
| WHERE container_cpu_usage_seconds_total IS NOT NULL OR kube_node_status_allocatable IS NOT NULL OR kube_pod_container_resource_limits IS NOT NULL
| STATS usage = SUM(CASE(((container != "" OR (container IS NULL AND "" != ""))), IRATE(container_cpu_usage_seconds_total), NULL)),
lhs = SUM(LAST_OVER_TIME(CASE((resource == "cpu") and (unit == "core"), kube_pod_container_resource_limits, NULL), 5m)),
rhs = SUM(LAST_OVER_TIME(CASE((resource == "cpu") and (unit == "core"), kube_node_status_allocatable, NULL), 5m))
BY time_bucket = TBUCKET(20, ?_tstart, ?_tend)
| STATS usage = MAX(usage), lhs = MAX(lhs), rhs = MAX(rhs)
```

## Impact

On [dashboard 15661](https://grafana.com/grafana/dashboards/15661-k8s-dashboard-en-20250125/):

| Panel | Grafana | Kibana |
|---|---|---|
| Node CPU Ratio | `46.6% / 41.5% / 83.0%` | `N/A / N/A / N/A` |
| Node CPU Number of cores | `20.0 / 6.7 / 8.3 / 16.6` | `N/A / N/A / N/A`, Usage only |

The memory counterparts (**Node Memory Ratio**, **Node Memory Information**) are correct — their usage term is a gauge, so the mix never occurs.

## Open question / suggested action

This may be an Elasticsearch ES|QL `TS` defect rather than a translator defect; the emitted ES|QL looks reasonable and F shows the same two aggregates coexisting happily without `CASE`. Two things seem worth doing regardless:

1. Confirm with the ES|QL team whether C/E are expected behaviour. If not, this needs an ES issue.
2. Until then, avoid emitting the failing shape — e.g. keep counter-rate targets in a separate `STATS`/subquery from `CASE`-filtered gauge targets, as in F.

A validation gate that flags "query succeeded but an aggregate column is entirely null" would have caught this at migrate time; today it passes as migrated.

## Minor, same panels

The gauge-tile lowering wraps values as `COALESCE(TO_STRING(x), "")` then `TO_DOUBLE(...)`, so a null column makes Elasticsearch emit a warning header on every refresh:

```
Line 10:62: org.elasticsearch.xpack.esql.core.InvalidArgumentException: Cannot parse number []
```

## Environment

- Engine: `main`
- Elasticsearch 9.5.0-SNAPSHOT / Kibana 9.5.0 (local `elastic-package` stack)
- Target layout: native `/_prometheus` remote-write data stream

Contributor guide

Open the contributing guide

Research direction

Start by reproducing query shapes C and E against the stated Elasticsearch 9.5.0-SNAPSHOT environment, then compare them with the working shapes F and G. Confirm with the ES|QL team whether the null aggregates are expected; done means the behavior is assigned to the correct component and either the failing shape is avoided or a validation gate flags silent all-null aggregate columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
backend, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.