huggingface / huggingface/transformers
Add SENet Blocks in Encoding Layers
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 276
Description
# 🚀 Feature Request
I read the article "[SesameBERT: Attention for Anywhere](https://arxiv.org/pdf/1910.03176.pdf)" and would like to add SENet blocks in the Huggingface implementation. The article's authors made an implementation with [Tensorflow](https://github.com/ICLR2020Sesame/SesameBert/blob/master/modeling.py), but I would like to use the lib in pytorch.
## Motivation
The use of ([Squeeze-and-Excitation Networks](https://arxiv.org/abs/1709.01507)) SENet Blocks has obtained state-of-the-art results. And they seem to be promising in NLP.
## Your contribution
I know that it is possible to modify the [[BertLayer()](https://github.com/huggingface/transformers/blob/61c506349134db0a0a2fd6fb2eff8e29a2f84e79/src/transformers/models/bert/modeling_bert.py#L430)] and [[BertEnconder()](https://github.com/huggingface/transformers/blob/61c506349134db0a0a2fd6fb2eff8e29a2f84e79/src/transformers/models/bert/modeling_bert.py#L513)] classes
Any suggestions on how to modify the code so that you can apply the idea used in the article?

Contributor guide
Assessment
This issue has not been assessed yet.