huggingface / huggingface/candle

Gemma models use the wrong activation function

Open Beginner friendly
#3,745 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Summary

`candle-transformers/src/models/quantized_gemma3.rs` hardcodes SiLU in the gated MLP:

```rust
let silu = candle_nn::ops::silu(&gate)?;
let gated = (silu * up)?;
self.feed_forward_down.forward(&gated)
```

Gemma 3's FFN activation is `gelu_pytorch_tanh`, not SiLU. Because GELU-tanh and SiLU are close in shape, nothing overflows and per-layer norms look healthy; the model just quietly produces lower-quality output, which is presumably why this has gone unnoticed.

Present in v0.11.0 and on current `main` (`candle-transformers/src/models/quantized_gemma3.rs`, the two lines in `impl Module for Mlp`).

Previously attempted in #3326; probably responsible for weirdness in #3492 and might be the root cause of #3299, as Gemma 4 uses the same activation function.

## Evidence that Gemma 3 is GELU-tanh

1. **HF transformers config default:** `src/transformers/models/gemma3/configuration_gemma3.py`:
```python
hidden_activation: str = "gelu_pytorch_tanh"
```
2. **llama.cpp:** `src/models/gemma3.cpp` builds the FFN with `LLM_FFN_GELU` (not `LLM_FFN_SILU`).
3. **candle itself:**the non-quantized `models/gemma3.rs` reads `cfg.hidden_activation` from the checkpoint config and therefore runs GELU-tanh at inference. The quantized and non-quantized implementations of the same model currently disagree.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in candle-transformers/src/models/quantized_gemma3.rs at impl Module for Mlp and compare its activation with the non-quantized models/gemma3.rs implementation. Check the referenced Hugging Face configuration and llama.cpp evidence, then verify that quantized Gemma 3 uses gelu_pytorch_tanh consistently with the non-quantized path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.