cloudflare / cloudflare/pint

Replace `count()` queries in `promql/series` with the Series Metadata API

Open
#1,796 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1k
Forks
70
Avg merge
1d 12h
Merged PRs (30d)
25

Description

`promql/series` validates metric existence by running bare `count()` queries. On high-cardinality metrics this results in long queries that can timeout and are inefficient.

## Example

For example, if the rule involves a traditionally high cardinality time series like `node_cpu_seconds_total`, it will run:

1. Instant query with labels: `count(node_cpu_seconds_total{mode="idle",job="node"})`
2. If step 1 returns nothing it goes to a bare range query of `count(node_cpu_seconds_total)` with all label matchers stripped which can be extremely expensive.

Step 2 is the biggest problem!

https://github.com/cloudflare/pint/blob/2a2e9d563aec8ed91515ea23f42437844f82afcb/internal/checks/promql_series.go#L247-L300

## Proposed Change

Replace both queries with a single call to the [`/api/v1/series`](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers) metadata endpoint which is pretty fast:

```
# labeled time series existence check
GET /api/v1/series?match[]=node_cpu_seconds_total{mode="idle",job="node"}&limit=1

# historical bare check (replaces range query)
GET /api/v1/series?match[]=node_cpu_seconds_total&limit=1&start=&end=
```

`limit=1` is also important as we only need the first match for this check, adding some extra efficiency gains. We verified on a large production instance:

```sh
# hangs with limit
curl '.../api/v1/series?match[]=node_cpu_seconds_total'

# returns immediately and stops on the first match
curl '.../api/v1/series?match[]=node_cpu_seconds_total&limit=1'
```

This should be a general improvement to the check with no downsides I can think of. As usual happy to contribute the change myself if you would like!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.