huggingface / huggingface/candle
Qwen/Qwen2-7B doesn't work properly in the example qwen
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
the Qwen/Qwen2-1.5B can work correct in the example, but Qwen/Qwen2-7B can't.
`(base) lyn@A100DEV:~/workspace/candle/candle-examples$ cargo run --release --features cuda --example qwen -- --model 2-7b --prompt "Hello\n"
Finished release [optimized] target(s) in 0.17s
Running `/disk/lyn/workspace/candle/target/release/examples/qwen --model 2-7b --prompt 'Hello\n'`
avx: true, neon: false, simd128: false, f16c: true
temp: 0.00 repeat-penalty: 1.10 repeat-last-n: 64
retrieved the files in 17.702388ms
loaded the model in 2.506053906s
Hello\n规范、、、、、、、 Orient Vall gu gap FourierbrtooltooltoolCharlesuketooltoolaurustooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltoolaurustoolrm哪些`
The model output is obviously incorrect.
the version:
commit 242e006bbb26ff12581b3c04bfd069996fe1f6bb (HEAD -> main, origin/main, origin/HEAD)
Author: Jeroen Vlek
Date: Mon Jun 24 19:12:52 2024 +0200
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the `qwen` example in `candle-examples` and reproduce the reported command using the CUDA release build. Compare its 7B behavior with the working 1.5B model and trace the model-loading or generation path; done means the 7B example produces coherent output rather than the shown repeated tokens.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100