allenai / allenai/dont-stop-pretraining
Add readme for the mlm study
- Lingua principale
- Python
- Stelle
- 544
- Fork
- 72
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
We want to report issues that could affect the reproducibility of the masked LM loss calculation at test time.
First, we do not get exactly the same results reported in Table 3 of paper when we use the fairseq library instead of the transformers library, after we convert the transformers checkpoint to a fairseq checkpoint.
A related pull [request](https://github.com/huggingface/transformers/pull/2928) was opened and closed, but did not fix our problem. Second, the results in Table 3 are calculated using the batch size of 1. With the batch sizes larger than 1, we do not get the same results. In particular, the results change for a sample of reviews. As we have already mentioned, reviews are much shorter than documents from other domains. Therefore, unlike documents in other domains that are usually of the maximum length, reviews need to be padded to the maximum length. For this reason, we suspect that padding somehow influences the masked LM loss calculation. However, with the batch size of 1 we do not need to pad, and therefore we find results in Table 3 reliable.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.