graykode / graykode/nlp-tutorial

Why is src_len+1 in Transformer demo?

Open
#66 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
14.9k
Forks
3.9k
PR merge metrics
No merged PRs in 30d

Description

self.pos_emb = nn.Embedding.from_pretrained(get_sinusoid_encoding_table(src_len+1, d_model),freeze=True)

The position encoding table should be (max_len, d_model), why add 1?

Contributor guide

Open the contributing guide

Research direction

Locate the Transformer demo containing the nn.Embedding.from_pretrained line and inspect get_sinusoid_encoding_table(src_len+1, d_model). Check how source positions are indexed and whether the extra position is required. Done means documenting a clear explanation of why src_len+1 is used, or confirming that the call should use src_len.

Written by the indexing model from the issue text.

Assessment

Tech stack
pytorch
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.