[Improve] benchmark ordered range scans and identify fan-out bottlenecks
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 636
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 14
Description
## Context
Follow-up to #3140. That PR establishes correctness for ordered HStore range-index scans through Store-local and client-side K-way merge. This issue is limited to measuring the performance characteristics and deciding which optimization, if any, is justified.
## Benchmark scope
Build a reproducible benchmark matrix covering:
- Small limits with many partitions
- Medium limits with sparse matches
- Large limits and page/cursor continuation
- 1, 4, 16, and 64 partitions
- Warm and cold cache, plus representative network latency
Record time to first result, total query time, RPC/open-stream count, transferred bytes, client heap usage, and Store CPU.
## Decisions after the baseline
Use the results to decide whether to open focused implementation issues for:
1. Partition pruning, when range metadata can safely eliminate non-overlapping partitions.
2. Client-driven initial and subsequent page-demand sizing, when it reduces unused data materially.
3. Prefetch/pipelining, after characterizing existing gRPC streaming and buffering behavior.
Store-local merge pushdown is already delivered by #3140 and is not duplicated here.
## Acceptance criteria
- Benchmark configuration and results are reproducible.
- The bottleneck and recommended next optimization are documented.
- Further implementation work is split into focused issues only when supported by the measurements.
I will follow up on this issue.
Contributor guide
Research direction
Start with the ordered HStore range-index scan behavior and the Store-local/client-side K-way merge established by #3140. Build the stated matrix across partition counts, cache and latency conditions, then record first-result and total time, RPCs, bytes, heap, and Store CPU; done means reproducible results, a documented bottleneck and recommendation, with follow-up implementation work split into focused issues.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- databases, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100