Query Mode that Ignores Uncompacted Segments
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
#### Is your feature request related to a problem? Please describe.
There is a high volume of segments before they are compacted into blocks - usually within a few minutes.
Segments span many tenants. Compacted blocks encapsulate a single tenant.
Queries reading segments require significantly more _work_ than blocks since they need to query a higher volume of segments. This means more operations to object storage. That translates to higher query latency and cloud costs (which bills you per object storage operation).
#### Describe the solution you'd like
Queries are faster and cheaper when they ignore segments and only query blocks.
I'd like to see support for a query mode that prevents segments from being queried.
As a server operator, I would actually prefer for this query mode to be the default and for segment queries to require an explicit opt-in via extra API parameter. That's because the query latency and cost delta penalty you pay to query the most recent few dozen seconds of data just isn't the right trade-off for my use cases.
#### Describe alternatives you've considered
Today, you need to teach clients to only query up to `now() -
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how server queries select segments and compacted blocks, then inspect the API paths used by profilecli and Profiles Drilldown. Define how the query mode is represented, whether block-only querying is the default, and how clients explicitly opt into segments; done means these behaviors are consistent and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend-api-design, cli, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100