Spill error (oom) with sort merge join
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
I run some data(300g) in small mem(64g) with sort merge join spill.
Then i get error
`Failed to allocate additional {} for {} with {} already allocated for this reservation - {} remain available for the total pool`
And i try to fix it. I fix two code
remove this code:
https://github.com/apache/datafusion/blob/c89cb70781dd3e8c52f7cbfff51c2b850900a132/datafusion/physical-plan/src/sorts/stream.rs#L176
and
https://github.com/apache/datafusion/blob/c89cb70781dd3e8c52f7cbfff51c2b850900a132/datafusion/physical-plan/src/sorts/sort.rs#L785
to
> loop {
> match self.reservation.try_grow(size) {
> Ok(_) => Ok(()),
> Err(e) => {
> if self.in_mem_batches.is_empty() {
> return Err(Self::err_with_oom_context(e));
> }
>
> // Spill and try again.
> self.sort_and_spill_in_mem_batches().await?;
> }
> }
I think the issues are as follows:
1. The memory was immediately occupied as soon as the sort operator spilled data to disk.
2. The row operator failed to spill data to disk, which caused this problem.
So could we fix it in this way, or are there any better solutions?
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reading the linked sections of datafusion/physical-plan/src/sorts/stream.rs and datafusion/physical-plan/src/sorts/sort.rs, focusing on reservation growth and spill handling. Reproduce the sort-merge join with the reported 300g workload and 64g of memory, then confirm that spilling completes without the reported reservation OOM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100