apache / apache/gluten

[VL] Use stream de/compressor in hash-based shuffle

Open
#10,075 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Hash-based shuffle may also have poor compression ratio, we should also use stream de/compressor for hash-based shuffle as in #9278.

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

Start by comparing the requested behavior with issue #9278 and locating the hash-based shuffle implementation and its existing compression tests. Confirm how stream de/compression is used there, then verify that hash-based shuffle uses the same approach and that relevant tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.