Explore batch-level refinement for spatial join query_batch
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 91
Description
Follow-up from #1026.
`query_batch()` currently refines candidates mostly at probe-row granularity. That works well when a single probe row produces enough candidates to trigger existing intra-row refinement parallelism, but it can underuse CPU when many probe rows each produce moderate candidate counts.
In #1026, I explored splitting probe rows into parallel chunks. That showed useful speedups on skewed workloads, but it also adds another parallelism layer/config knob and overlaps conceptually with `ProbeShuffleExec` / partition-level parallelism.
An alternative worth exploring is batch-level refinement:
- collect candidates across multiple probe rows
- trigger refinement once a candidate threshold is reached
- preserve row-major output and existing `max_result_size` / resume semantics
- reuse or align with the existing `parallel_refinement_chunk_size` behavior if possible
Suggested benchmark comparisons:
- current per-row refinement
- partition-level parallelism via `ProbeShuffleExec`
- probe-row chunking from #1026 as a reference point
- candidate accumulation across rows
This should help decide whether the right fix is a new probe-row parallel path, a refinement batching change, or simply better benchmark coverage around the existing partition-level behavior.
Contributor guide
Research direction
Start with query_batch() and the existing parallel_refinement_chunk_size behavior. Compare current per-row refinement with ProbeShuffleExec, the probe-row chunking from #1026, and candidate accumulation across rows using the suggested benchmarks. Done means identifying whether batch-level refinement, a new probe-row parallel path, or better partition-level benchmark coverage is appropriate while preserving row-major output, max_result_size, and resume semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100