huggingface / huggingface/candle
Phi-3 implementation seems to be buggy on metal devices
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
After running multiple times the command:
```
cargo run --release --features metal --example phi -- --model 3 --prompt "The best thing about coding in rust is "
```
I realized a very degrading performance in the token generation time, on my Macbook Pro M3. After profiling the issue, realized that with a `repeat_penalty = 1.1`, there is roughly (on avg) 50secs spent on
```rs
candle_transformers::utils::apply_repeat_penalty(
&logits,
self.repeat_penalty,
&tokens[start_at..],
)?
```
and notably, the time taken is mostly on the following:
```rs
let mut logits = logits.to_dtype(candle::DType::F32)?.to_vec1::()?;
```
I find this weird, as my Macbook Pro M3 is very fast for a Mamba 2.8b model, which is roughly the size of Phi-3 mini. Also the above operation is an allocation of a data buffer of roughly 30/40 thousand `f32`'s, which is definitely not that large.
As a follow-up question, a few lines above, I see that the forward pass is done through:
```rs
Model::Phi3(m) => m.forward(&input, pos)?.i((.., 0, ..))?,
```
which might have an impact on the display of the tensor and therefore it might affect the allocation. I can keep investigating though.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.