allenai / allenai/longformer

local vs global attention in further MLM pre-training.

未关闭
#180 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
2.2k
派生
286
PR 合并指标
30 天内没有已合并 PR

描述

I am currently looking into the [conversion script](https://colab.research.google.com/github/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) to create a longformer from an already pre-trained Roberta implementation. I noticed that the script further pre-trains the Longformer on MLM with only local attention.

- What exactly is the motive behind not having tokens (e.g. bos and eos) attend globally as well?
- Would it make sense to alter this if the goal you have in mind is focussed on tasks that require embeddings of the entire sequence? (e.g. unsupervised document similarity)

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。