[VL] Looks CelebornShuffleReader doesn't spill data
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
```c++
25/05/28 17:12:50 ERROR [Executor task launch worker for task 0.0 in stage 399.0 (TID 76139)] ManagedReservationListener: Error reserving memory from target
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 6.0 MiB. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=5.6 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=1433.5 MiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=716.8 MiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats:
Task.76139: Current used bytes: 5.6 GiB, peak bytes: N/A
+- Gluten.Tree.5: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| \- Capacity[8.0 EiB].5: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| +- CelebornShuffleReader.5: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| | \- single: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| | +- root: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| | | \- default_leaf: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| | \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- ArrowContextInstance.5: Current used bytes: 8.0 MiB, peak bytes: 8.0 MiB
| +- ColumnarToRow.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 2.4 MiB
| +- ColumnarToRow.0: Current used bytes: 0.0 B, peak bytes: 8.0 MiB
| | \- single: Current used bytes: 0.0 B, peak bytes: 8.0 MiB
| | +- root: Current used bytes: 0.0 B, peak bytes: 7.0 MiB
| | | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 7.0 MiB
| | \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- CelebornShuffleReader.5.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 1288.8 MiB
\- org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@4d16a67d: Current used bytes: 136.0 MiB, peak bytes: N/A
```
### Gluten version
Gluten-1.3
### Spark version
Spark-3.2.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start at the CelebornShuffleReader entry point and compare its memory-consumer statistics with the reported OutOfMemoryException. Reproduce the Spark 3.2.x, Gluten 1.3 scenario using the listed off-heap settings, then verify whether shuffle data spills and document or fix the behavior indicated by the memory usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100