mars-project / mars-project/mars

Support work stealing

Open
#486 0 comments 0 reactions 0 assignees View on GitHub
type: enhancement
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

We may encounter slow nodes when executing on multiple workers. The initial assignment algorithm is not guaranteed to be optimal. The execution graph may have a larger output set than input set. Hence work stealing is important to further accelerate execution.

The implementation of work stealing should be based on #485 .

Contributor guide

Open the contributing guide

Research direction

Start by reading issue #485 and tracing the execution graph and multi-worker scheduling described there. Determine how work is assigned when the output set exceeds the input set, then define completion as work stealing being implemented and improving execution across slow and otherwise idle workers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.