Support for different hardware configurations for different task roles of one distributed job.
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 2.7k
- Forks
- 554
- Avg merge
- 20h 42m
- Merged PRs (30d)
- 14
Description
What would you like to be added:
Support for different hardware configurations for different task roles of one distributed job.
Why is this needed:
For complex learning tasks, the programs that need to run on each computer are very different, and the requirements for CPU /GPU and RAM /GPU memory are also different. At the same time, these computers need to communicate with each other to enable joint training. For example, in reinforcement learning, the entire reinforcement learning algorithm consists of different modules. The actor uses the GPU to generate data, the learner uses the GPU to train data, the environment and MCTS use CPU to generate data in parallel, and these modules involve complex data communication.
Without this feature, how does the current module work:
Reinforcement learning tasks cannot be performed jointly by multiple computers.
Components that may involve changes:
Job protocol and related.
Downgrade vc to taskrole:

Allows each taskrole to have a different skutype:

Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the job protocol and the existing taskrole and skutype handling mentioned in the issue. Determine how one distributed job currently represents hardware requirements, then define support for distinct configurations per task role and verify that reinforcement-learning-style roles can communicate and schedule correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, kubernetes
- Domain
- distributed-systems, infrastructure, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100