apache / apache/gluten

[VL] Distinct aggregation OOM when getOutput

Open
#8,025 21 comments 3 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Distinct aggregation will merge all sorted spill file in `getOutput()` (`SpillPartition::createOrderedReader`). If there are too many spill files, reading the first batch of each file into memory will consume a significant amount of memory. In one of our internal cases, one task generated 300 spill files, which requires close to 3G of memory.

![image](https://github.com/user-attachments/assets/23dd540e-a4b7-448e-84e0-caae00aa5147)

Possible workarounds:

1. Increase `kMaxSpillRunRows`, `1M` will generate too many spill files for hundreds million rows of input. https://github.com/apache/incubator-gluten/pull/7531
2. Reduce `kSpillWriteBufferSize` to `1M` or lower. Why it is set to 4M by default? Is there any experience in performance tuning?

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.