Confusing attention_window configuration in converting roberta-base to longformer notebook
- Lingua principale
- Python
- Stelle
- 2.2k
- Fork
- 285
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
Hi.
I try to follow `scripts/convert_model_to_long.ipynb` to pretrain longformer from roberta-base, the default number of attention_window in ModelArgs make bugs,
```
@dataclass
class ModelArgs:
attention_window: int = field(default=512, metadata={"help": "Size of attention window"})
max_pos: int = field(default=4096, metadata={"help": "Maximum position"})
parser = HfArgumentParser((TrainingArguments, ModelArgs,))
```
attention_window = 512 make LongformerSelfAttention's actual window size to be 1024, after which I cannot reproduce the third row in tab 6 in [paper](https://arxiv.org/pdf/2004.05150.pdf).
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.