deepspeedai / deepspeedai/DeepSpeed
Hooks to add sharding and parallelism for custom layers (Not Conv1D or Linear)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Hi, I've been trying to find where in DeepSpeed one would go about adding sharding and parallelism for a custom layer, that has more than 1 input.
https://www.deepspeed.ai/training/ lists Support for Custom Model Parallelism on it's feature overview, but the header above it (Model Parallelism) links to https://www.deepspeed.ai/features/#model-parallelism
Which results in a 404.
Is there any external hooks or guides on how to go about this?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the training page and its Model Parallelism link, which currently leads to a 404, then inspect how DeepSpeed exposes support for custom model parallelism. Done means providing documented hooks or guidance for sharding and parallelizing custom layers with multiple inputs, beyond Conv1D and Linear.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100