allenai / allenai/longformer

should both: attention_mask and global_attention_mask be used for classification?

Open
#75 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.2k
Forks
285
PR merge metrics
No merged PRs in 30d

Description

Hi,

Again a conceptual question on text classification.

Since global attention is used on `````` only, I am slightly confused if I should just pass global_attention_mask to the model or both: attention_mask and global_attention_mask. I follow that attention_mask is mainly used to mask the `````` tokens but does it mean ``` n ``` 2 complexity for local attention?

Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.