Global attention in key_padding_mask
未关闭
- 主要语言
- Python
- 星标
- 2.2k
- 派生
- 285
- PR 合并指标
- 30 天内没有已合并 PR
描述
Hi,
I'm trying to understand the attention_mask=key_padding_mask.unsqueeze(dim=1).unsqueeze(dim=1) * -1 in class LongformerSelfAttentionForBart. 'key_padding_mask' has only True or False as far as I understand due to
"def invert_mask(attention_mask):
"""Turns 1->0, 0->1, False->True, True-> False"""
assert attention_mask.dim() == 2
return attention_mask.eq(0)"
in modeling_bart.py. It would be great if you could point me to the part of the code how it adds global attention while we go through modeling_bart for Longformer+Bart.
Thanks,
Akila
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。