mars-project / mars-project/mars

Ray-based Shuffle For Mars

Open
#2,916 3 comments 2 reactions 0 assignees View on GitHub
proposal
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

This issue is used for tracking Ray-based Shuffle For Mars progress:
- [x] Ray Future-based Shuffle MEP:https://github.com/mars-project/meps/pull/2
- [x] Shuffle Meta optimization #3055
- [x] Ray Future-based Shuffle
- [x] Basic Ray Shuffle Implementation #3040
- [x] Shuffle object GC
- [x] Shuffle operands compatibility
- [ ] Push-based Ray Shuffle
- [ ] Push-based ray shuffle MEP
- [ ] Scheduling all ray tasks in placement group
- [ ] Pipelined push-based Shuffle
- [ ] Push strategy optimization. Skip push shuffle blocks whose size exceed specified size.
- [ ] Support configure merge factor.
- [ ] Support configure concurrent running mapper subtasks.
- [ ] Support adjust merge factor adaptively
- [ ] Support adjust concurrent running mapper subtasks adaptively to reduce write amplification
- [ ] Handle shuffle slow sutbask by speculative execution
- [ ] Dynamic merge shuffle partitions
- [ ] Dynamic split shuffle partitions
- [ ] Dynamic detect skewed partitions
- [ ] Dynamic optimize skewed join
- [ ] Support sub chunks
[ ] Shuffle Failover based lineage reconstruction

Contributor guide

Open the contributing guide

Research direction

This is a long-running tracking issue for Ray-based Shuffle in Mars, with completed work and several broad unchecked tasks. Start by reviewing the linked Ray Future-based Shuffle MEP and referenced issues #3055 and #3040, then identify one remaining item with a defined scope. Done would require completing that item and updating the checklist.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.