codertimo / codertimo/BERT-pytorch
when training the masked LM, the unmasked words (have label 0) were trained together with masked words?
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
According to the code
```
def random_word(self, sentence):
tokens = sentence.split()
output_label = []
for i, token in enumerate(tokens):
prob = random.random()
if prob < 0.15:
# 80% randomly change token to make token
if prob < prob * 0.8:
tokens[i] = self.vocab.mask_index
# 10% randomly change token to random token
elif prob * 0.8 <= prob < prob * 0.9:
tokens[i] = random.randrange(len(self.vocab))
# 10% randomly change token to current token
elif prob >= prob * 0.9:
tokens[i] = self.vocab.stoi.get(token, self.vocab.unk_index)
output_label.append(self.vocab.stoi.get(token, self.vocab.unk_index))
else:
tokens[i] = self.vocab.stoi.get(token, self.vocab.unk_index)
output_label.append(0)
return tokens, output_label
```
Do we need to exclude the unmasked words when training the LM?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.