huggingface / huggingface/candle
Quantized Phi-3 example fails "cannot find llama.attention.head_count in metadata"
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
```
cargo run --example quantized-phi --release -- --prompt "what is the best thing about rust?" --which phi-3
Compiling candle-examples v0.5.0 (/Users/kenk/Documents/Code/OpenSource/candle/candle-examples)
Finished release [optimized] target(s) in 3.21s
Running `target/release/examples/quantized-phi --prompt 'what is the best thing about rust?' --which phi-3`
avx: false, neon: true, simd128: false, f16c: false
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
Running on CPU, to run on GPU(metal), build this example with `--features metal`
loaded 195 tensors (2.39GB) in 0.08s
Error: cannot find llama.attention.head_count in metadata
```
hardware: M1 macbook
This issue does not occur when using phi-2 or with any other example that I've tried. It also still occurs even with the `metal` feature enabled.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.