apache / apache/sedona-db

Explore batch-level refinement for spatial join query_batch

Open
#1,054 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 5h
Merged PRs (30d)
91

Description

Follow-up from #1026.

`query_batch()` currently refines candidates mostly at probe-row granularity. That works well when a single probe row produces enough candidates to trigger existing intra-row refinement parallelism, but it can underuse CPU when many probe rows each produce moderate candidate counts.

In #1026, I explored splitting probe rows into parallel chunks. That showed useful speedups on skewed workloads, but it also adds another parallelism layer/config knob and overlaps conceptually with `ProbeShuffleExec` / partition-level parallelism.

An alternative worth exploring is batch-level refinement:

- collect candidates across multiple probe rows
- trigger refinement once a candidate threshold is reached
- preserve row-major output and existing `max_result_size` / resume semantics
- reuse or align with the existing `parallel_refinement_chunk_size` behavior if possible

Suggested benchmark comparisons:

- current per-row refinement
- partition-level parallelism via `ProbeShuffleExec`
- probe-row chunking from #1026 as a reference point
- candidate accumulation across rows

This should help decide whether the right fix is a new probe-row parallel path, a refinement batching change, or simply better benchmark coverage around the existing partition-level behavior.

Contributor guide

Open the contributing guide

Research direction

Start with query_batch() and the existing parallel_refinement_chunk_size behavior. Compare current per-row refinement with ProbeShuffleExec, the probe-row chunking from #1026, and candidate accumulation across rows using the suggested benchmarks. Done means identifying whether batch-level refinement, a new probe-row parallel path, or better partition-level benchmark coverage is appropriate while preserving row-major output, max_result_size, and resume semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.