codertimo / codertimo/BERT-pytorch
why language_model.py has different vectors
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
In language_model.py codes, class NextSentencePrediction and class MaskedLanguageModel has different input x in their forward function, In class NextSentencePrediction the input use "x[: 0] in "return self.softmax(self.linear(x[:, 0]))" but in class MaskedLanguageModel the input use x in "return self.softmax(self.linear(x))", I think if there is something wrong?
As I debug to this, both x in their has shape of (batch_size, seq_len, embedding_dim) like (64, 50, 256), we known this mains I have 64 sentences and each sentences has 50 words and each word is a 256 dim of vector. But the x[:, 0] means I take the first word in every 64 sentences so lead x[:, 0] to has shape of (64, 256). I don't understand why the task NextSentencePrediction should use this kind of input, can someone help me to explain this?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.