mars-project / mars-project/mars
Support work stealing
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
We may encounter slow nodes when executing on multiple workers. The initial assignment algorithm is not guaranteed to be optimal. The execution graph may have a larger output set than input set. Hence work stealing is important to further accelerate execution.
The implementation of work stealing should be based on #485 .
Contributor guide
Research direction
Start by reading issue #485 and tracing the execution graph and multi-worker scheduling described there. Determine how work is assigned when the output set exceeds the input set, then define completion as work stealing being implemented and improving execution across slow and otherwise idle workers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100