NestedLoopJoinExec spill fallback evaluates the left input twice
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
## Problem
When `NestedLoopJoinExec` cannot fully buffer its left input, the spill fallback executes the left child again:
1. The normal path calls `left.execute(0)` and starts collecting the stream.
2. Collection fails with `ResourcesExhausted`.
3. The fallback calls `left.execute(0)` again to create the spill input.
The shared spill state prevents every probe partition from repeating the scan, but the left input is still evaluated twice overall.
This behavior was discussed during review of #21448:
- https://github.com/apache/datafusion/pull/21448#discussion_r3063018913
- https://github.com/apache/datafusion/pull/21448#discussion_r3068909276
This is a focused follow-up to #15760 and #21448.
## Why this matters
Repeated evaluation can duplicate expensive scans, network reads, or other work performed by the left subtree. It also increases pressure during an OOM recovery path.
This issue does not propose changing `ExecutionPlan::execute` semantics. Each call should continue producing an independent stream, as established by #21565. The goal is to avoid the redundant call where possible.
## Proposed direction
Refactor the initial left-side collection so that, when its reservation cannot grow, the fallback can retain:
- Already buffered batches.
- The batch that encountered memory pressure.
- The remaining original input stream.
The fallback could then spill or process that state directly instead of executing the left child again.
Alternative designs that guarantee a single left-side evaluation would also address the issue.
## Testing
Add a test execution plan that:
- Counts calls to `execute()`.
- Produces an independent stream on every call.
- Is used as the left child of `NestedLoopJoinExec`.
- Forces the memory-limited spill path.
Assert that:
- The left child is executed exactly once.
- Spill occurs.
- The query returns the expected results.
- Memory reservations and temporary spill files are released after completion or cancellation.
## References
- Parent tracking issue: #15760
- Initial implementation: #21448
- Commit: `f9239a197f2c49b69a87efb5ef25fd67b13b7ca1`
- Independent stream semantics: #21565
Contributor guide
Research direction
Start at NestedLoopJoinExec's initial left-side collection and spill fallback, tracing how buffered batches, the pressure-triggering batch, and the remaining stream are handled. Add the execution-plan test described in the issue, then verify one left-child execute call, spill, expected results, and cleanup after completion or cancellation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100