prometheus / prometheus/prometheus

Using `Select()` for both sample+metadata queries is unsuitable for remote storage implementations

Open
#4,057 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/api component/remote storage help wanted priority/P3
Dominant language
Go
Stars
66.1k
Forks
10.8k
Avg merge
2d 1h
Merged PRs (30d)
131

Description

The storage.Querier interface has a Select() method which can be used to retrieve either only metadata or bulk sample data of series. In the local storage case, this works nicely, as sample data is loaded lazily when accessed, and thus metadata-only queries never over-fetch sample data. This breaks down for remote storage implementations of the same interface, where we cannot afford to lazily fetch sample data only as it's accessed (that would require more remote round trips). We need some way of knowing beforehand (when we call Select()) whether a query is metadata-only or whether it actually needs sample data. Either a separate method or query parameter would work.

Context: Cortex reuses Prometheus's web API and PromQL packages to offer comparable functionality based on a different storage engine. In the past, we were able to solve this issue because the Prometheus web API packages allowed injecting two different queriers: one that would be used as part of the PromQL engine (the full sample querier) and one that would be used for metadata queries. This got broken in https://github.com/prometheus/prometheus/commit/7ccd4b39b89e936bf0f9c47811e3fe2b20d1fe3b#diff-d81f5cda89ea7b129ba708b586c2bc83L132, due to a PromQL engine restructuring (the querier is not part of the engine anymore, it gets passed in for every query, and the web API only knows about one querier now, because we cannot encapsulate the other one in the engine anymore).

The issue about this on the Cortex side is https://github.com/weaveworks/cortex/issues/787

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the storage.Querier.Select() callers and the Prometheus web API/PromQL integration described in the issue. Determine where metadata-only and bulk sample queries are distinguished, then verify that remote implementations can avoid lazy sample fetches while both query paths remain supported for Cortex.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.