codertimo / codertimo/BERT-pytorch
why specify `ignore_index=0` in the NLLLoss function in BERTTrainer?
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
# trainer/pretrain.py
```python
class BERTTrainer:
def __init__(self, ...):
...
# Using Negative Log Likelihood Loss function for predicting the masked_token
self.criterion = nn.NLLLoss(ignore_index=0)
...
```
I cannot understand why `ignore index=0` is specified when calculating NLLLoss. If the ground truth of `is_next` is False (label = 0) in terms of the NSP task but BERT predicts True, then NLLLoss will be 0 (or nan)... so what's the aim of `ignore_index = 0` ???
====================
Well, I've found that `ignore_index = 0` is useful to the MLM task, but I still can't agree the NSP task should share the same NLLLoss with MLM.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.