Add MemoryReservation to batch splitting in joins
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Follow on to https://github.com/apache/datafusion/pull/12969 and https://github.com/apache/datafusion/issues/12633
In https://github.com/apache/datafusion/issues/12633 @mhilton noted that joins sometimes generate giant record batches which causes issues. @alihan-synnada fixed this in https://github.com/apache/datafusion/pull/12969 but internally sometimes the joins still generate giant output batches.
As @mhilton says in https://github.com/apache/datafusion/pull/12969#issuecomment-2418862655
> Unfortunately this doesn't address the actual problem with creating giant batches, which is they require a lot of memory and that memory isn't accounted for in any MemoryPool. Wiring a MemoryReservation into BatchSplitter would probably be enough to address this though.
### Describe the solution you'd like
I would like the memory accounting to take into account the large output batch
### Describe alternatives you've considered
Wiring a MemoryReservation into BatchSplitter would probably be enough to address
### Additional context
_No response_
Contributor guide
Research direction
Start by reviewing BatchSplitter and the linked pull request 12969 and issue 12633 to understand how join output batches are split. Trace where the output batch's memory is created and determine how MemoryReservation should cover it; done means the large output batch is accounted for in a MemoryPool.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100