deepspeedai / deepspeedai/DeepSpeed
Converting DeepSpeedTransformerLayer to Hugging Face Transformers' BERTLayer
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Hi, thank you for publicizing this amazing work!
When pre_layer_norm=False, I think that the DeepSpeedTransformerLayer is exactly equivalent to the vanilla hidden layer of the transformer architecture. If so, is it possible to convert DeepSpeedTransformerLayer to BERTLayer in Hugging Face Transformers?
It seems that DeepSpeedTransformerLayer has the following parameters: 'attn_qkvw', 'attn_qkvb', 'attn_ow', 'attn_ob', 'attn_nw', 'attn_nb', 'inter_w', 'inter_b', 'output_w', 'output_b', 'norm_w', 'norm_b'. When assigning them to the corresponding weights in BERTLayer, can we obtain the same model?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing DeepSpeedTransformerLayer with Hugging Face Transformers' BERTLayer, using the listed parameters as the mapping checklist. Determine whether assigning those weights produces an equivalent model, and document the conversion steps or any incompatibilities found.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100