apache / apache/gluten

[VL] Hash shuffle may leave local spill payloads unmerged when partition-buffer eviction writes pids out of order

Open
#12,235 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

Hash shuffle can fail during ShuffleWriter.stop() with:
```
Native shuffle write: ShuffleWriter stop failed - Invalid:
Merging from spill N is not exhausted. pid: X
```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start at ShuffleWriter.stop() and trace the hash-shuffle path involving partition-buffer eviction and local spill merging. Use the reported “Merging from spill N is not exhausted” failure as the symptom to reproduce or follow; done means stop succeeds without leaving spill payloads unmerged when pids are written out of order.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.