huggingface / huggingface/candle

Phi-3 implementation seems to be buggy on metal devices

Open
#2,128 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

After running multiple times the command:

```
cargo run --release --features metal --example phi -- --model 3 --prompt "The best thing about coding in rust is "
```

I realized a very degrading performance in the token generation time, on my Macbook Pro M3. After profiling the issue, realized that with a `repeat_penalty = 1.1`, there is roughly (on avg) 50secs spent on

```rs
candle_transformers::utils::apply_repeat_penalty(
&logits,
self.repeat_penalty,
&tokens[start_at..],
)?
```

and notably, the time taken is mostly on the following:

```rs
let mut logits = logits.to_dtype(candle::DType::F32)?.to_vec1::()?;
```

I find this weird, as my Macbook Pro M3 is very fast for a Mamba 2.8b model, which is roughly the size of Phi-3 mini. Also the above operation is an allocation of a data buffer of roughly 30/40 thousand `f32`'s, which is definitely not that large.

As a follow-up question, a few lines above, I see that the forward pass is done through:

```rs
Model::Phi3(m) => m.forward(&input, pos)?.i((.., 0, ..))?,
```

which might have an impact on the display of the tensor and therefore it might affect the allocation. I can keep investigating though.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.