Replace `count()` queries in `promql/series` with the Series Metadata API
- Dominant language
- Go
- Stars
- 1k
- Forks
- 70
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 25
Description
`promql/series` validates metric existence by running bare `count()` queries. On high-cardinality metrics this results in long queries that can timeout and are inefficient.
## Example
For example, if the rule involves a traditionally high cardinality time series like `node_cpu_seconds_total`, it will run:
1. Instant query with labels: `count(node_cpu_seconds_total{mode="idle",job="node"})`
2. If step 1 returns nothing it goes to a bare range query of `count(node_cpu_seconds_total)` with all label matchers stripped which can be extremely expensive.
Step 2 is the biggest problem!
https://github.com/cloudflare/pint/blob/2a2e9d563aec8ed91515ea23f42437844f82afcb/internal/checks/promql_series.go#L247-L300
## Proposed Change
Replace both queries with a single call to the [`/api/v1/series`](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers) metadata endpoint which is pretty fast:
```
# labeled time series existence check
GET /api/v1/series?match[]=node_cpu_seconds_total{mode="idle",job="node"}&limit=1
# historical bare check (replaces range query)
GET /api/v1/series?match[]=node_cpu_seconds_total&limit=1&start=&end=
```
`limit=1` is also important as we only need the first match for this check, adding some extra efficiency gains. We verified on a large production instance:
```sh
# hangs with limit
curl '.../api/v1/series?match[]=node_cpu_seconds_total'
# returns immediately and stops on the first match
curl '.../api/v1/series?match[]=node_cpu_seconds_total&limit=1'
```
This should be a general improvement to the check with no downsides I can think of. As usual happy to contribute the change myself if you would like!
Contributor guide
Assessment
This issue has not been assessed yet.