graykode / graykode/nlp-tutorial
Why is src_len+1 in Transformer demo?
Open
- Dominant language
- Jupyter Notebook
- Stars
- 14.9k
- Forks
- 3.9k
- PR merge metrics
- No merged PRs in 30d
Description
self.pos_emb = nn.Embedding.from_pretrained(get_sinusoid_encoding_table(src_len+1, d_model),freeze=True)
The position encoding table should be (max_len, d_model), why add 1?
Contributor guide
Assessment
This issue has not been assessed yet.