Replace `record_batch.get_array_memory_size()` in spilling operators
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
This issue is trying to track the follow on tasks for https://github.com/apache/datafusion/pull/13377
1. Currently record batches' memory size will be overestimated, and this issue will be (partly) fixed by https://github.com/apache/datafusion/pull/13377 After merge, other usage of `record_batch.get_array_memory_size()` should be replaced (e.g. Memory counting in `TopK` and `Sort-Merge-Join`) After that, more end-to-end tests for that specific operator can be added
2. Check whether known memory-limited query related bug can be fixed by '1'. Issues maybe related are:
https://github.com/apache/datafusion/issues/13089
https://github.com/apache/datafusion/issues/9417
https://github.com/apache/datafusion/issues/10511
https://github.com/apache/datafusion/issues/12136
https://github.com/apache/datafusion/issues/11390
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reviewing pull request 13377 and locating the remaining record_batch.get_array_memory_size() uses in the TopK and Sort-Merge-Join spilling operators. Check the related memory-limited query issues, then add operator-specific end-to-end tests; done means memory accounting no longer uses the overestimating method in those paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100