huggingface / huggingface/candle

DeBERTa-v2 models fail to run at F16

Open Beginner friendly
#3,750 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Problem

`DebertaV2Model::forward` fails for F16 models with:

```
dtype mismatch in add, lhs: F32, rhs: F16
```

The same model and inputs work at F32. This blocks running DeBERTa-v2 (and models built on it, e.g. GLiNER2) at reduced precision — which is what makes a DeBERTa-v2-scale encoder fit in memory on wasm32.

## Reproduction

Verified against candle 0.11.0; the offending lines are unchanged on current `main`.

```rust
use candle_core::{DType, Device, Tensor};
use candle_nn::VarBuilder;
use candle_transformers::models::debertav2::{Config, DebertaV2Model};

fn main() {
let dev = Device::Cpu;
let config: Config = serde_json::from_value(serde_json::json!({
"vocab_size": 32, "hidden_size": 16, "num_hidden_layers": 1,
"num_attention_heads": 2, "intermediate_size": 32,
"hidden_act": "gelu", "hidden_dropout_prob": 0.0,
"attention_probs_dropout_prob": 0.0, "max_position_embeddings": 32,
"type_vocab_size": 0, "initializer_range": 0.02,
"layer_norm_eps": 1e-7, "relative_attention": true,
"max_relative_positions": -1, "pad_token_id": 0,
"position_biased_input": true,
"pos_att_type": ["p2c", "c2p"], "position_buckets": 8, "norm_rel_ebd": "layer_norm"
})).unwrap();
for dtype in [DType::F32, DType::F16] {
let vb = VarBuilder::zeros(dtype, &dev);
let model = DebertaV2Model::load(vb, &config).unwrap();
let ids = Tensor::zeros((1, 6), DType::U32, &dev).unwrap();
match model.forward(&ids, None, None) {
Ok(out) => println!("{dtype:?}: OK {:?}", out.dtype()),
Err(e) => println!("{dtype:?}: ERR {e}"),
}
}
}
```

Output:

```
F32: OK F32
F16: ERR dtype mismatch in add, lhs: F32, rhs: F16
```

(BF16 fails earlier with `unsupported dtype BF16 for op matmul`, a separate limitation.)

## Root cause

`debertav2.rs` hardcodes F32 scalars in paths that must run at the model's working dtype:

1. `disentangled_attention_bias` initializes its `score` accumulator as F32; it is then `broadcast_add`ed with the F16 attention terms — the site of the error above.
2. `XSoftmax::apply` builds both mask fill tensors as F32.

## Fix

Cast the scalars to the working dtype with `.to_dtype(...)`. With exactly that change, the reproduction prints `F16: OK F16` and a real GLiNER2 (DeBERTa-v2 encoder) runs at F16 in the browser.

Found by @jamon8888 in a PR bringing in-browser DeBERTa-v2 inference to [xberg](https://github.com/xberg-io/xberg); minimized and re-verified against stock 0.11.0 for this report.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in debertav2.rs at disentangled_attention_bias and XSoftmax::apply, then run the provided minimal reproduction for F32 and F16. Done means the reproduction reports F16: OK F16 and the dtype mismatch no longer occurs in the DeBERTa-v2 forward path.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, wasm
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.