local vs global attention in further MLM pre-training.
- Lingua principale
- Python
- Stelle
- 2.2k
- Fork
- 285
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
I am currently looking into the [conversion script](https://colab.research.google.com/github/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) to create a longformer from an already pre-trained Roberta implementation. I noticed that the script further pre-trains the Longformer on MLM with only local attention.
- What exactly is the motive behind not having tokens (e.g. bos and eos) attend globally as well?
- Would it make sense to alter this if the goal you have in mind is focussed on tasks that require embeddings of the entire sequence? (e.g. unsupervised document similarity)
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.