[VL] Distinct aggregation OOM when getOutput
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
Distinct aggregation will merge all sorted spill file in `getOutput()` (`SpillPartition::createOrderedReader`). If there are too many spill files, reading the first batch of each file into memory will consume a significant amount of memory. In one of our internal cases, one task generated 300 spill files, which requires close to 3G of memory.

Possible workarounds:
1. Increase `kMaxSpillRunRows`, `1M` will generate too many spill files for hundreds million rows of input. https://github.com/apache/incubator-gluten/pull/7531
2. Reduce `kSpillWriteBufferSize` to `1M` or lower. Why it is set to 4M by default? Is there any experience in performance tuning?
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.