deepspeedai / deepspeedai/DeepSpeed
[QUESTION] GPT2 not supported by AutoTP
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Hi,
We are recently enabling Megatron-Deepspeed Inference(GPT2 model) on the CPU and want to use AutoTP to reduce latency. Can I ask the reason that AutoTP cannot support GPT2? Is it because of the FusedLayerNorm/ParralleLinear module of GPT2 or due to other reasons?
Appreciate if I can get a brief answer from you. We wonder whether we can add autoTP support to GPT2？Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names AutoTP, GPT2, FusedLayerNorm, and ParallelLinear but no files or tests. Start by tracing AutoTP's GPT2 handling and comparing those modules with supported models; done means documenting the incompatibility or defining the support work needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100