apache / apache/datafusion

Spill error (oom) with sort merge join

Open
#19,013 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

I run some data(300g) in small mem(64g) with sort merge join spill.
Then i get error
`Failed to allocate additional {} for {} with {} already allocated for this reservation - {} remain available for the total pool`

And i try to fix it. I fix two code
remove this code:
https://github.com/apache/datafusion/blob/c89cb70781dd3e8c52f7cbfff51c2b850900a132/datafusion/physical-plan/src/sorts/stream.rs#L176

and

https://github.com/apache/datafusion/blob/c89cb70781dd3e8c52f7cbfff51c2b850900a132/datafusion/physical-plan/src/sorts/sort.rs#L785

to

> loop {
> match self.reservation.try_grow(size) {
> Ok(_) => Ok(()),
> Err(e) => {
> if self.in_mem_batches.is_empty() {
> return Err(Self::err_with_oom_context(e));
> }
>
> // Spill and try again.
> self.sort_and_spill_in_mem_batches().await?;
> }
> }

I think the issues are as follows:
1. The memory was immediately occupied as soon as the sort operator spilled data to disk.
2. The row operator failed to spill data to disk, which caused this problem.
So could we fix it in this way, or are there any better solutions?

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked sections of datafusion/physical-plan/src/sorts/stream.rs and datafusion/physical-plan/src/sorts/sort.rs, focusing on reservation growth and spill handling. Reproduce the sort-merge join with the reported 300g workload and 64g of memory, then confirm that spilling completes without the reported reservation OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.