[VL] Use stream de/compressor in hash-based shuffle
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Hash-based shuffle may also have poor compression ratio, we should also use stream de/compressor for hash-based shuffle as in #9278.
### Gluten version
None
Contributor guide
Research direction
Start by comparing the requested behavior with issue #9278 and locating the hash-based shuffle implementation and its existing compression tests. Confirm how stream de/compression is used there, then verify that hash-based shuffle uses the same approach and that relevant tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100