LayerNorm eps value
- Dominant language
- Python
- Stars
- 4.3k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
Hi!
thanks for this little piece of juicy code!
Just for curiosity, I've noticed that in your implementation you are using `nn.LayerNorm` with the standard denominator constant `eps=1e-5`, whereas in other implementations (`DINO` [[here](https://github.com/facebookresearch/dino/blob/main/vision_transformer.py)] and `ViT` in `timm`[[here](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py)]) this parameter is explicitly set to `eps=1e-6`.
I know that it is a small detail, but details sometimes are super-important for having better models.
Do you think the model is sensitive to this kind of parameter change? Have you ever tried/noticed it?
Thanks!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.