deepspeedai / deepspeedai/DeepSpeed
[REQUEST] Support Deepspeed inference for Fairseq Transformer LM model
Open
@awan-10 is already working on this.
Since Mar 13, 2023.
enhancement
inference
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
We would like to use Deepspeed Inference engine for drawing inference on Fairseq based Transformer LM model. Currently we run into error while loading the deepspeed trained checkpoint.
Describe the solution you'd like
Support kernel injections for the Fairseq Transformer LM model trained using Deepspeed.
Describe alternatives you've considered
We can also make modifications in this model architecture or implementation if it helps to get the transofmer_lm to work out of box with deepspeed inference in a similar way for gpt models
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.