huggingface / huggingface/candle
Mamba model is broken with `f16` precision
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
Running the command:
```cargo run --example mamba --release --features metal -- --prompt "Tell me a joke please" --dtype f16```
does not work. The problem seems to lie in code:
```rs
for &t in tokens.iter() {
let input = Tensor::new(&[t], &self.device)?;
let logits = self.model.forward(&input, &mut state)?;
next_logits = Some(logits);
if let Some(t) = self.tokenizer.next_token(t)? {
print!("{t}")
}
}
```
where `logits` is a `Tensor` of `null` values.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the `cargo run --example mamba --release --features metal -- --prompt "Tell me a joke please" --dtype f16` entry point and inspect the token loop shown in the issue. Trace why `self.model.forward` produces null logits for f16; done means the command generates output with non-null logits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100