deepspeedai / deepspeedai/DeepSpeed
Distillation implementation for 3D parallelism.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Hi,
I am looking for implementing GPT model for distillation but I found that for pipeline parallelism the model should be using LayerSpec which is nn.sequential like. Is there any workaround to make it work?
Distillation is a model compression technique whereby a small network (student) is taught by a larger trained neural network (teacher).
"pseudocode"
model_input = (input_ids, mask, position_ids)
t_output = teacher_model(model_input )
s_output = student_model(model_input )
loss = t_output - s_output
Student_model.optimize(loss)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by investigating LayerSpec and pipeline parallelism in the GPT model path, then determine what a working teacher-student distillation workflow across 3D parallelism would require; done means the implementation supports that workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100