apache / apache/gluten

[VL] Looks CelebornShuffleReader doesn't spill data

Open
#9,784 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

```c++
25/05/28 17:12:50 ERROR [Executor task launch worker for task 0.0 in stage 399.0 (TID 76139)] ManagedReservationListener: Error reserving memory from target
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 6.0 MiB. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled).
Current config settings:
spark.gluten.memory.offHeap.size.in.bytes=5.6 GiB
spark.gluten.memory.task.offHeap.size.in.bytes=1433.5 MiB
spark.gluten.memory.conservative.task.offHeap.size.in.bytes=716.8 MiB
spark.memory.offHeap.enabled=true
spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats:
Task.76139: Current used bytes: 5.6 GiB, peak bytes: N/A
+- Gluten.Tree.5: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| \- Capacity[8.0 EiB].5: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| +- CelebornShuffleReader.5: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| | \- single: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| | +- root: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| | | \- default_leaf: Current used bytes: 5.5 GiB, peak bytes: 5.5 GiB
| | \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| +- ArrowContextInstance.5: Current used bytes: 8.0 MiB, peak bytes: 8.0 MiB
| +- ColumnarToRow.0.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 2.4 MiB
| +- ColumnarToRow.0: Current used bytes: 0.0 B, peak bytes: 8.0 MiB
| | \- single: Current used bytes: 0.0 B, peak bytes: 8.0 MiB
| | +- root: Current used bytes: 0.0 B, peak bytes: 7.0 MiB
| | | \- default_leaf: Current used bytes: 0.0 B, peak bytes: 7.0 MiB
| | \- gluten::MemoryAllocator: Current used bytes: 0.0 B, peak bytes: 0.0 B
| \- CelebornShuffleReader.5.OverAcquire.0: Current used bytes: 0.0 B, peak bytes: 1288.8 MiB
\- org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@4d16a67d: Current used bytes: 136.0 MiB, peak bytes: N/A
```

### Gluten version

Gluten-1.3

### Spark version

Spark-3.2.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start at the CelebornShuffleReader entry point and compare its memory-consumer statistics with the reported OutOfMemoryException. Reproduce the Spark 3.2.x, Gluten 1.3 scenario using the listed off-heap settings, then verify whether shuffle data spills and document or fix the behavior indicated by the memory usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.