mars-project / mars-project/mars
Ray-based Shuffle For Mars
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
This issue is used for tracking Ray-based Shuffle For Mars progress:
- [x] Ray Future-based Shuffle MEP:https://github.com/mars-project/meps/pull/2
- [x] Shuffle Meta optimization #3055
- [x] Ray Future-based Shuffle
- [x] Basic Ray Shuffle Implementation #3040
- [x] Shuffle object GC
- [x] Shuffle operands compatibility
- [ ] Push-based Ray Shuffle
- [ ] Push-based ray shuffle MEP
- [ ] Scheduling all ray tasks in placement group
- [ ] Pipelined push-based Shuffle
- [ ] Push strategy optimization. Skip push shuffle blocks whose size exceed specified size.
- [ ] Support configure merge factor.
- [ ] Support configure concurrent running mapper subtasks.
- [ ] Support adjust merge factor adaptively
- [ ] Support adjust concurrent running mapper subtasks adaptively to reduce write amplification
- [ ] Handle shuffle slow sutbask by speculative execution
- [ ] Dynamic merge shuffle partitions
- [ ] Dynamic split shuffle partitions
- [ ] Dynamic detect skewed partitions
- [ ] Dynamic optimize skewed join
- [ ] Support sub chunks
[ ] Shuffle Failover based lineage reconstruction
Contributor guide
Research direction
This is a long-running tracking issue for Ray-based Shuffle in Mars, with completed work and several broad unchecked tasks. Start by reviewing the linked Ray Future-based Shuffle MEP and referenced issues #3055 and #3040, then identify one remaining item with a defined scope. Done would require completing that item and updating the checklist.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100