codertimo / codertimo/BERT-pytorch

why specify `ignore_index=0` in the NLLLoss function in BERTTrainer?

Open
#98 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
6.5k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

# trainer/pretrain.py

```python
class BERTTrainer:
def __init__(self, ...):
...
# Using Negative Log Likelihood Loss function for predicting the masked_token
self.criterion = nn.NLLLoss(ignore_index=0)
...
```

I cannot understand why `ignore index=0` is specified when calculating NLLLoss. If the ground truth of `is_next` is False (label = 0) in terms of the NSP task but BERT predicts True, then NLLLoss will be 0 (or nan)... so what's the aim of `ignore_index = 0` ???

====================

Well, I've found that `ignore_index = 0` is useful to the MLM task, but I still can't agree the NSP task should share the same NLLLoss with MLM.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.