codertimo / codertimo/BERT-pytorch

when training the masked LM, the unmasked words (have label 0) were trained together with masked words?

Open
#29 6 comments 0 reactions 0 assignees View on GitHub
enhancement question
Dominant language
Python
Stars
6.5k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

According to the code

```
def random_word(self, sentence):
tokens = sentence.split()
output_label = []

for i, token in enumerate(tokens):
prob = random.random()
if prob < 0.15:
# 80% randomly change token to make token
if prob < prob * 0.8:
tokens[i] = self.vocab.mask_index

# 10% randomly change token to random token
elif prob * 0.8 <= prob < prob * 0.9:
tokens[i] = random.randrange(len(self.vocab))

# 10% randomly change token to current token
elif prob >= prob * 0.9:
tokens[i] = self.vocab.stoi.get(token, self.vocab.unk_index)

output_label.append(self.vocab.stoi.get(token, self.vocab.unk_index))

else:
tokens[i] = self.vocab.stoi.get(token, self.vocab.unk_index)
output_label.append(0)

return tokens, output_label
```

Do we need to exclude the unmasked words when training the LM?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.