graykode / graykode/nlp-tutorial
Why is src_len+1 in Transformer demo?
Open
- Dominant language
- Jupyter Notebook
- Stars
- 14.9k
- Forks
- 3.9k
- PR merge metrics
- No merged PRs in 30d
Description
self.pos_emb = nn.Embedding.from_pretrained(get_sinusoid_encoding_table(src_len+1, d_model),freeze=True)
The position encoding table should be (max_len, d_model), why add 1?
Contributor guide
Research direction
Locate the Transformer demo containing the nn.Embedding.from_pretrained line and inspect get_sinusoid_encoding_table(src_len+1, d_model). Check how source positions are indexed and whether the extra position is required. Done means documenting a clear explanation of why src_len+1 is used, or confirming that the call should use src_len.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pytorch
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100