Question about MLP
Open
- Dominant language
- Python
- Stars
- 4.3k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
[Here](https://github.com/karpathy/makemore/blob/988aa59e4d8fefa526d06f3b453ad116258398d4/makemore.py#L382) you are padding the tensor with special starting token. It looks strange to me that you are doing it inside the embedding. Isn't this strange?
Aren't you supposed to first pass the special token through the embedding first and then add that as a padding?
```
tok_emb = self.wte(idx) # token embeddings of shape (b, t, n_embd)
idx = torch.roll(idx, 1, 1)
# something like this instead?
idx[:, 0] = self.wte(self.vocab_size) # special token
embs.append(tok_emb)
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.