huggingface / huggingface/candle
Gemma models use the wrong activation function
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Summary
`candle-transformers/src/models/quantized_gemma3.rs` hardcodes SiLU in the gated MLP:
```rust
let silu = candle_nn::ops::silu(&gate)?;
let gated = (silu * up)?;
self.feed_forward_down.forward(&gated)
```
Gemma 3's FFN activation is `gelu_pytorch_tanh`, not SiLU. Because GELU-tanh and SiLU are close in shape, nothing overflows and per-layer norms look healthy; the model just quietly produces lower-quality output, which is presumably why this has gone unnoticed.
Present in v0.11.0 and on current `main` (`candle-transformers/src/models/quantized_gemma3.rs`, the two lines in `impl Module for Mlp`).
Previously attempted in #3326; probably responsible for weirdness in #3492 and might be the root cause of #3299, as Gemma 4 uses the same activation function.
## Evidence that Gemma 3 is GELU-tanh
1. **HF transformers config default:** `src/transformers/models/gemma3/configuration_gemma3.py`:
```python
hidden_activation: str = "gelu_pytorch_tanh"
```
2. **llama.cpp:** `src/models/gemma3.cpp` builds the FFN with `LLM_FFN_GELU` (not `LLM_FFN_SILU`).
3. **candle itself:**the non-quantized `models/gemma3.rs` reads `cfg.hidden_activation` from the checkpoint config and therefore runs GELU-tanh at inference. The quantized and non-quantized implementations of the same model currently disagree.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in candle-transformers/src/models/quantized_gemma3.rs at impl Module for Mlp and compare its activation with the non-quantized models/gemma3.rs implementation. Check the referenced Hugging Face configuration and llama.cpp evidence, then verify that quantized Gemma 3 uses gelu_pytorch_tanh consistently with the non-quantized path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100