codertimo / codertimo/BERT-pytorch
chooses 15% of token
Open
good first issue
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
From paper, it mentioned
> Instead, the training data generator chooses 15% of tokens at random, e.g., in the sentence my
> dog is hairy it chooses hairy.
It means that 15% of token will be choose for sure.
From https://github.com/codertimo/BERT-pytorch/blob/master/bert_pytorch/dataset/dataset.py#L68,
for every single token, it has 15% of chance that go though the followup procedure. Does it aligned with 15% of token will be chosen?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.