BERT存在两个严重问题

1.随机替换成词表中的其他词时应该排除[PAD],[CLS],[SEP],[MASK]，否则预测模型会学会预测这些词。
2.学习率开大了，始终难以收敛，开1e-6很合适。

The issue identifies BERT's random token replacement and learning-rate settings but names no file or test. Start by locating the BERT training notebook and the token-replacement and optimizer configuration; done means special tokens are excluded from replacement and the learning rate is 1e-6, with training behavior checked.

Written by the indexing model from the issue text.