mars-project / mars-project/mars
Use queues instead of direct allocation in workers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
Currently, Mars worker allocates CPUs for tasks via ``DispatchActor``, who allocates CPUs in an exclusive manner, that is, when a CPU is allocated to a task, it is removed from a pool. There is no problem with this mechanism when calculation is started immediately. However, currently in Mars, we load data which cannot be loaded into shared memory into process memory before actually starting calculation. This can waste CPU resource when data are loaded into process memory.
We may use a new CPU allocation strategy by introducing queues for slots instead of allocating them directly. When a task is queued, it starts loading data into a calculation process if needed, and waits for the process to be ready before actually starts execution. The queued tasks who finish data preparation are executed first.
Possible influences of this strategy on existing quota mechanism should be considered. What's more, whether this mechanism works in real-world tasks should be studied as well.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the worker allocation path around DispatchActor to understand the current exclusive CPU mechanism. Study how queued slots would interact with data preparation and existing quota behavior. Done means the proposed queue strategy is validated for task ordering, resource use, quota effects, and real-world workloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100