huggingface / huggingface/candle

Mamba model is broken with `f16` precision

Open
#2,280 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Running the command:

```cargo run --example mamba --release --features metal -- --prompt "Tell me a joke please" --dtype f16```

does not work. The problem seems to lie in code:

```rs
for &t in tokens.iter() {
let input = Tensor::new(&[t], &self.device)?;
let logits = self.model.forward(&input, &mut state)?;
next_logits = Some(logits);
if let Some(t) = self.tokenizer.next_token(t)? {
print!("{t}")
}
}
```

where `logits` is a `Tensor` of `null` values.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the `cargo run --example mamba --release --features metal -- --prompt "Tell me a joke please" --dtype f16` entry point and inspect the token loop shown in the issue. Trace why `self.model.forward` produces null logits for f16; done means the command generates output with non-null logits.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.