allenai / allenai/longformer

local vs global attention in further MLM pre-training.

Aperta
#180 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
2.2k
Fork
285
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

I am currently looking into the [conversion script](https://colab.research.google.com/github/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) to create a longformer from an already pre-trained Roberta implementation. I noticed that the script further pre-trains the Longformer on MLM with only local attention.

- What exactly is the motive behind not having tokens (e.g. bos and eos) attend globally as well?
- Would it make sense to alter this if the goal you have in mind is focussed on tasks that require embeddings of the entire sequence? (e.g. unsupervised document similarity)

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.