apache / apache/datafusion

Add MemoryReservation to batch splitting in joins

Open
#13,003 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Follow on to https://github.com/apache/datafusion/pull/12969 and https://github.com/apache/datafusion/issues/12633

In https://github.com/apache/datafusion/issues/12633 @mhilton noted that joins sometimes generate giant record batches which causes issues. @alihan-synnada fixed this in https://github.com/apache/datafusion/pull/12969 but internally sometimes the joins still generate giant output batches.

As @mhilton says in https://github.com/apache/datafusion/pull/12969#issuecomment-2418862655

> Unfortunately this doesn't address the actual problem with creating giant batches, which is they require a lot of memory and that memory isn't accounted for in any MemoryPool. Wiring a MemoryReservation into BatchSplitter would probably be enough to address this though.

### Describe the solution you'd like

I would like the memory accounting to take into account the large output batch

### Describe alternatives you've considered

Wiring a MemoryReservation into BatchSplitter would probably be enough to address

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing BatchSplitter and the linked pull request 12969 and issue 12633 to understand how join output batches are split. Trace where the output batch's memory is created and determine how MemoryReservation should cover it; done means the large output batch is accounted for in a MemoryPool.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.