elastic / elastic/observability-migration-platform
Grafana: PromQL regex with angle brackets (`!~"<none>|Job"`) emits an invalid ES|QL RLIKE pattern
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 23
Description
## Summary
A PromQL negative-regex matcher whose alternation contains angle brackets — e.g. `created_by_kind!~"|Job"` — is translated verbatim into an ES|QL `RLIKE` pattern. ES|QL `RLIKE` uses Lucene regexp syntax, where `<...>` is the **numeric range** operator, so the pattern is rejected and the panel fails to render at all.
`` is the standard `kube-state-metrics` value for "no owner", so this affects Kubernetes dashboards generally.
## Observed error
In the uploaded Kibana dashboard the panel shows an error instead of a chart:
```
Couldn't parse Elasticsearch ES|QL query. Check your query and try again.
Error: line 4:56: Invalid regex pattern for RLIKE [|Job]: ['none' not found]
```
## Reproduction
Source PromQL (from [dashboard 15661](https://grafana.com/grafana/dashboards/15661-k8s-dashboard-en-20250125/)):
```promql
count(kube_pod_info{created_by_kind!~"|Job", node=~"^$Node$"})
```
Emitted ES|QL (excerpt):
```esql
TS metrics-k8s.prometheus-parity
| WHERE kube_node_info IS NOT NULL OR kube_pod_info IS NOT NULL OR kube_node_status_allocatable IS NOT NULL
| STATS Pod_Number_of_nodes = COUNT(CASE(((NOT (created_by_kind RLIKE "|Job")
OR (created_by_kind IS NULL AND NOT ("" RLIKE "|Job")))) and (...), kube_pod_info, NULL)), ...
```
## Impact
Two panels on dashboard 15661 hard-fail (no render at all, error card):
- **Nodes with Pod** — Grafana shows `3 / 15 / 330`
- **Node Information Detail** — the node inventory table
Both are graded `migrated_with_warnings` / Yellow; neither warning mentions the regex.
## Root cause
ES|QL `RLIKE` follows Lucene regexp syntax, in which `<` and `>` delimit a numeric range (e.g. `<1-100>`). The Prometheus/RE2 pattern `|Job` is valid RE2 but invalid Lucene, so `'none'` is parsed as a malformed range bound.
## Suggested fix
When lowering a PromQL `=~` / `!~` matcher to `RLIKE`, escape (or otherwise neutralise) the Lucene regexp metacharacters that RE2 treats as literals — at minimum `<` and `>`, and worth auditing `#`, `@`, `&`, `~` for the same class of mismatch.
Alternatively, when the pattern is a plain alternation of literals (the common case), emit `IN (...)` / `NOT IN (...)` instead of `RLIKE`, which sidesteps regex-dialect differences entirely and is faster.
Live ES|QL validation should also catch this at migrate time: the run reported these panels as migrated, and the failure only appeared in the browser.
## Environment
- Engine: `main`
- Elasticsearch 9.5.0-SNAPSHOT / Kibana 9.5.0 (local `elastic-package` stack)
- Target layout: native `/_prometheus` remote-write data stream (`metrics-k8s.prometheus-parity`)
Contributor guide
Research direction
Start by locating the PromQL =~/!~ lowering that emits ES|QL RLIKE and the migration-time validation path; reproduce the matcher from dashboard 15661. Check Lucene/RE2 metacharacter handling, especially < and >, and audit the plain-alternation alternative. Done means the reproduced panels no longer fail and migration validation catches invalid patterns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana, python
- Domain
- backend, observability, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100