Memory accounting inaccuracies in multi level merge
Open
bug
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Not enough memory is reserved for the SortPreservingMerge
* track the record batch buffering (even with buffer_size=1 there's one record batch being prefetched by spawn_buffer)
* take into account that for the minimum requirement for two stream merge, we need to reserve 3 record batches because there could be a record batch crossing happening inside one of the streams
* for multi-column sort, take into account the additional overhead of ReusableRows
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.