AnswerDotAI / AnswerDotAI/ModernBERT
First layer residual is inconsistently normalized
Open
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
For most configurations, the first encoder blocks' pre-norm is skipped due to the embedding layers' post-normalization.
Yet, having this "pseudo-prenorm" in the embedding layer also means that the residual taken in that first block will be normalized.
This is inconsistent with how pre-norm is originally formulated, where the norm is *not* applied to the residual (https://arxiv.org/pdf/2002.04745, Fig. 1).
Any particular reason why it is implemented differently in ModernBERT?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.