huggingface / huggingface/candle
DeBERTa-v2 models fail to run at F16
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Problem
`DebertaV2Model::forward` fails for F16 models with:
```
dtype mismatch in add, lhs: F32, rhs: F16
```
The same model and inputs work at F32. This blocks running DeBERTa-v2 (and models built on it, e.g. GLiNER2) at reduced precision — which is what makes a DeBERTa-v2-scale encoder fit in memory on wasm32.
## Reproduction
Verified against candle 0.11.0; the offending lines are unchanged on current `main`.
```rust
use candle_core::{DType, Device, Tensor};
use candle_nn::VarBuilder;
use candle_transformers::models::debertav2::{Config, DebertaV2Model};
fn main() {
let dev = Device::Cpu;
let config: Config = serde_json::from_value(serde_json::json!({
"vocab_size": 32, "hidden_size": 16, "num_hidden_layers": 1,
"num_attention_heads": 2, "intermediate_size": 32,
"hidden_act": "gelu", "hidden_dropout_prob": 0.0,
"attention_probs_dropout_prob": 0.0, "max_position_embeddings": 32,
"type_vocab_size": 0, "initializer_range": 0.02,
"layer_norm_eps": 1e-7, "relative_attention": true,
"max_relative_positions": -1, "pad_token_id": 0,
"position_biased_input": true,
"pos_att_type": ["p2c", "c2p"], "position_buckets": 8, "norm_rel_ebd": "layer_norm"
})).unwrap();
for dtype in [DType::F32, DType::F16] {
let vb = VarBuilder::zeros(dtype, &dev);
let model = DebertaV2Model::load(vb, &config).unwrap();
let ids = Tensor::zeros((1, 6), DType::U32, &dev).unwrap();
match model.forward(&ids, None, None) {
Ok(out) => println!("{dtype:?}: OK {:?}", out.dtype()),
Err(e) => println!("{dtype:?}: ERR {e}"),
}
}
}
```
Output:
```
F32: OK F32
F16: ERR dtype mismatch in add, lhs: F32, rhs: F16
```
(BF16 fails earlier with `unsupported dtype BF16 for op matmul`, a separate limitation.)
## Root cause
`debertav2.rs` hardcodes F32 scalars in paths that must run at the model's working dtype:
1. `disentangled_attention_bias` initializes its `score` accumulator as F32; it is then `broadcast_add`ed with the F16 attention terms — the site of the error above.
2. `XSoftmax::apply` builds both mask fill tensors as F32.
## Fix
Cast the scalars to the working dtype with `.to_dtype(...)`. With exactly that change, the reproduction prints `F16: OK F16` and a real GLiNER2 (DeBERTa-v2 encoder) runs at F16 in the browser.
Found by @jamon8888 in a PR bringing in-browser DeBERTa-v2 inference to [xberg](https://github.com/xberg-io/xberg); minimized and re-verified against stock 0.11.0 for this report.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in debertav2.rs at disentangled_attention_bias and XSoftmax::apply, then run the provided minimal reproduction for F32 and F16. Done means the reproduction reports F16: OK F16 and the dtype mismatch no longer occurs in the DeBERTa-v2 forward path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, wasm
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100