elastic / elastic/observability-migration-platform

Grafana: PromQL regex with angle brackets (`!~"<none>|Job"`) emits an invalid ES|QL RLIKE pattern

Open
#371 0 comments 0 reactions 0 assignees View on GitHub
asset:dashboards bug program:grafana-engine source:grafana
Dominant language
Python
Stars
6
Forks
8
Avg merge
2d 22h
Merged PRs (30d)
23

Description

## Summary

A PromQL negative-regex matcher whose alternation contains angle brackets — e.g. `created_by_kind!~"|Job"` — is translated verbatim into an ES|QL `RLIKE` pattern. ES|QL `RLIKE` uses Lucene regexp syntax, where `<...>` is the **numeric range** operator, so the pattern is rejected and the panel fails to render at all.

`` is the standard `kube-state-metrics` value for "no owner", so this affects Kubernetes dashboards generally.

## Observed error

In the uploaded Kibana dashboard the panel shows an error instead of a chart:

```
Couldn't parse Elasticsearch ES|QL query. Check your query and try again.
Error: line 4:56: Invalid regex pattern for RLIKE [|Job]: ['none' not found]
```

## Reproduction

Source PromQL (from [dashboard 15661](https://grafana.com/grafana/dashboards/15661-k8s-dashboard-en-20250125/)):

```promql
count(kube_pod_info{created_by_kind!~"|Job", node=~"^$Node$"})
```

Emitted ES|QL (excerpt):

```esql
TS metrics-k8s.prometheus-parity
| WHERE kube_node_info IS NOT NULL OR kube_pod_info IS NOT NULL OR kube_node_status_allocatable IS NOT NULL
| STATS Pod_Number_of_nodes = COUNT(CASE(((NOT (created_by_kind RLIKE "|Job")
OR (created_by_kind IS NULL AND NOT ("" RLIKE "|Job")))) and (...), kube_pod_info, NULL)), ...
```

## Impact

Two panels on dashboard 15661 hard-fail (no render at all, error card):

- **Nodes with Pod** — Grafana shows `3 / 15 / 330`
- **Node Information Detail** — the node inventory table

Both are graded `migrated_with_warnings` / Yellow; neither warning mentions the regex.

## Root cause

ES|QL `RLIKE` follows Lucene regexp syntax, in which `<` and `>` delimit a numeric range (e.g. `<1-100>`). The Prometheus/RE2 pattern `|Job` is valid RE2 but invalid Lucene, so `'none'` is parsed as a malformed range bound.

## Suggested fix

When lowering a PromQL `=~` / `!~` matcher to `RLIKE`, escape (or otherwise neutralise) the Lucene regexp metacharacters that RE2 treats as literals — at minimum `<` and `>`, and worth auditing `#`, `@`, `&`, `~` for the same class of mismatch.

Alternatively, when the pattern is a plain alternation of literals (the common case), emit `IN (...)` / `NOT IN (...)` instead of `RLIKE`, which sidesteps regex-dialect differences entirely and is faster.

Live ES|QL validation should also catch this at migrate time: the run reported these panels as migrated, and the failure only appeared in the browser.

## Environment

- Engine: `main`
- Elasticsearch 9.5.0-SNAPSHOT / Kibana 9.5.0 (local `elastic-package` stack)
- Target layout: native `/_prometheus` remote-write data stream (`metrics-k8s.prometheus-parity`)

Contributor guide

Open the contributing guide

Research direction

Start by locating the PromQL =~/!~ lowering that emits ES|QL RLIKE and the migration-time validation path; reproduce the matcher from dashboard 15661. Check Lucene/RE2 metacharacter handling, especially < and >, and audit the plain-alternation alternative. Done means the reproduced panels no longer fail and migration validation catches invalid patterns.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, python
Domain
backend, observability, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.