deepspeedai / deepspeedai/DeepSpeedExamples
The example of bert compression did not change line 383 in modeling_bert.py?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
In example of model compression, it says only change is line 383 "where we output attention_scores instead of attention_prob.". But this line is the same as hugging face and I think it does not output attention_scores. Am i wrong or is there a typo?
By the way if only one line need to be changed, is it possible to apply deepspeed compression on deberta v2(hugging face https://github.com/huggingface/transformers/blob/main/src/transformers/models/deberta_v2/modeling_deberta_v2.py)?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Open model_compression/bert/huggingface_transformer/modeling_bert.py at line 383 and compare it with the referenced Hugging Face implementation. Verify whether the example actually outputs attention_scores, then check the referenced DeBERTa v2 modeling file for the corresponding behavior. Done means correcting the example or documenting the discrepancy and clarifying whether the compression approach applies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100