bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed
DeBERTa-like attention mechanism
Open
arch&scale
enhancement
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 226
- PR merge metrics
- No merged PRs in 30d
Description
In this issue, we discuss how viable/interesting it might be to implement DeBERTa like attention mechanism:
https://arxiv.org/abs/2006.03654
Things to take in account:
- performance enhancements: Check with HF pretrained model to see first?
- implementation cost: How much would someone need to spend on implementing that feature?
- implementation feasability: It might not work well with Megatron-DeepSpeed setup, we need to check that.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.