Roadmap of Mars Distributed Engine

Open

#493 opened on Jun 26, 2019 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language: Python
Stars: 2.7k
Forks: 325
PR merge metrics: No merged PRs in 30d

Description

New Functions
---------------
* [x] (M4) All-or-nothing scheduler for existing systems such as XGBoost (#270)
* [ ] (M?) Support for user-defined operands (TBD)
* [ ] (L6) Support for iterative tiling (#184)

Efficiency
----------
* [ ] **(H4)** Metric subsystem (#485)
* [ ] (H2) Support for estimating queueing time
* [ ] (H1) Support for finish-time estimation
* [ ] (L1) Support for computational costs of non-deterministic operands
* [ ] **(H?)** Optimization for huge graphs (#106)
* [ ] (M2) New job queueing mechanism (#363)
* [ ] **(H5)** Work stealing mechanism (#486)
* [ ] (L6) Broadcasting protocols (#489)
* [ ] Peer-to-peer approach
* [ ] **(H?)** More aggressive scheduling policy (TBD)

Hardware
----------
* [x] **(H4)** Support for GPU (#302)
* [ ] Refine host memory allocation
* [ ] Support spill when device memory is full
* [ ] Further CUDA-specific optimizations

Resilience
----------
* [x] **(H3)** Fail-over for shuffle (#484)
* [ ] (M3) Automatic memory-size tuning for crashed operands (TBD)
* [ ] (M6) Fail-over for multiple graphs (#239)
* [ ] (L4) Checkpointing (TBD)
* [ ] (??) Automatic checkpointing

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Research direction

This is a broad roadmap with no files, tests, or entry points identified. Start by reviewing the linked issues, especially #485, #486, and #489, and select one unchecked item with a defined scope. Done cannot be determined until that item is split into a concrete issue with implementation locations and acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack: python
Domain: distributed-systems
Issue type: Feature
Difficulty: 5/5
Estimated time: Over a week
Activity status: Stale
Clarity: Needs clarification
Newbie friendliness: 15/100