apache / apache/hugegraph

[Improve] benchmark ordered range scans and identify fan-out bottlenecks

Open
#3,158 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
636
Avg merge
3d 11h
Merged PRs (30d)
14

Description

## Context

Follow-up to #3140. That PR establishes correctness for ordered HStore range-index scans through Store-local and client-side K-way merge. This issue is limited to measuring the performance characteristics and deciding which optimization, if any, is justified.

## Benchmark scope

Build a reproducible benchmark matrix covering:

- Small limits with many partitions
- Medium limits with sparse matches
- Large limits and page/cursor continuation
- 1, 4, 16, and 64 partitions
- Warm and cold cache, plus representative network latency

Record time to first result, total query time, RPC/open-stream count, transferred bytes, client heap usage, and Store CPU.

## Decisions after the baseline

Use the results to decide whether to open focused implementation issues for:

1. Partition pruning, when range metadata can safely eliminate non-overlapping partitions.
2. Client-driven initial and subsequent page-demand sizing, when it reduces unused data materially.
3. Prefetch/pipelining, after characterizing existing gRPC streaming and buffering behavior.

Store-local merge pushdown is already delivered by #3140 and is not duplicated here.

## Acceptance criteria

- Benchmark configuration and results are reproducible.
- The bottleneck and recommended next optimization are documented.
- Further implementation work is split into focused issues only when supported by the measurements.

I will follow up on this issue.

Contributor guide

Open the contributing guide

Research direction

Start with the ordered HStore range-index scan behavior and the Store-local/client-side K-way merge established by #3140. Build the stated matrix across partition counts, cache and latency conditions, then record first-result and total time, RPCs, bytes, heap, and Store CPU; done means reproducible results, a documented bottleneck and recommendation, with follow-up implementation work split into focused issues.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.