huggingface / huggingface/transformers

Add SENet Blocks in Encoding Layers

Open
#11,998 1 comment 0 reactions 0 assignees View on GitHub
Feature request
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 8h
Merged PRs (30d)
276

Description

# 🚀 Feature Request

I read the article "[SesameBERT: Attention for Anywhere](https://arxiv.org/pdf/1910.03176.pdf)" and would like to add SENet blocks in the Huggingface implementation. The article's authors made an implementation with [Tensorflow](https://github.com/ICLR2020Sesame/SesameBert/blob/master/modeling.py), but I would like to use the lib in pytorch.

## Motivation

The use of ([Squeeze-and-Excitation Networks](https://arxiv.org/abs/1709.01507)) SENet Blocks has obtained state-of-the-art results. And they seem to be promising in NLP.

## Your contribution

I know that it is possible to modify the [[BertLayer()](https://github.com/huggingface/transformers/blob/61c506349134db0a0a2fd6fb2eff8e29a2f84e79/src/transformers/models/bert/modeling_bert.py#L430)] and [[BertEnconder()](https://github.com/huggingface/transformers/blob/61c506349134db0a0a2fd6fb2eff8e29a2f84e79/src/transformers/models/bert/modeling_bert.py#L513)] classes

Any suggestions on how to modify the code so that you can apply the idea used in the article?

![image](https://user-images.githubusercontent.com/25322394/120578580-0f3ca000-c3f4-11eb-9a14-93bf610c3a60.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.