Gemma: A study of the effect of the new issues

Nobody has claimed this yet.

[Daniel Han](https://twitter.com/danielhanchen) in his [blogpost](https://unsloth.ai/blog/gemma-bugs) shared his discoveries of what is ~also~ wrong with the Gemma implementation.
Some of them are only appliable to Keras, some of them affects PyTorch too.
This issue will include information concerning those bugs and ours codebase.

- [ ] 1. Must add \
- [ ] 2. There’s a typo for \model
- [x] 3. $sqrt(3072)=55.4256$ but bfloat16 is $55.5$
- [ ] 4. Layernorm (w+1) must be in float32
- [ ] 5. Keras mixed_bfloat16 RoPE is wrong
- [ ] 6. RoPE is sensitive to $y \cdot \left(\frac{1}{x}\right)$ vs $y/x$
- [ ] 7. RoPE should be float32
- [x] 8. GELU should be approx tanh not exact
- [ ] 9. Additional: check why tests for LoRA and AdapterV2 variants of GemmaMLP fail

Feel free to chime in.

Thanks to @kashif for letting us know.

Start by reading the linked Gemma bug report and compare its findings with the Gemma implementation. Work through the unchecked checklist items, including the LoRA and AdapterV2 GemmaMLP test failures; done means the applicable implementation issues are corrected and the affected tests pass.

Written by the indexing model from the issue text.