huggingface / huggingface/transformers
Implementing ELECTRIC training for ELECTRA
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 276
Description
# 🚀 Feature request
Google released Electric this summer at EMNLP (see: [here](https://www.aclweb.org/anthology/2020.emnlp-main.20.pdf)). Electric is like ELECTRA, but trained using a Noise Contrastive Estimation loss instead of a negative sampling loss.
## Motivation
Electric is well-suited for modeling perplexity scores, and can model these very efficiently. Modeling these perplexity scores using BERT requires N passes over the input sentence, where N is the number of tokes in the sentence (see [here](https://arxiv.org/abs/1910.14659)).
## Your contribution
Electric has been implemented in the Google Electra repository. From I can see, moving from an Electra to Electric-style training is not a huge code change, but I'm not that familiar with the inner workings of transformers to be able to make a judgment call on this.
Contributor guide
Assessment
This issue has not been assessed yet.