local vs global attention in further MLM pre-training.
未关闭
- 主要语言
- Python
- 星标
- 2.2k
- 派生
- 286
- PR 合并指标
- 30 天内没有已合并 PR
描述
I am currently looking into the [conversion script](https://colab.research.google.com/github/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) to create a longformer from an already pre-trained Roberta implementation. I noticed that the script further pre-trains the Longformer on MLM with only local attention.
- What exactly is the motive behind not having tokens (e.g. bos and eos) attend globally as well?
- Would it make sense to alter this if the goal you have in mind is focussed on tasks that require embeddings of the entire sequence? (e.g. unsupervised document similarity)
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。