huggingface / huggingface/candle

Qwen/Qwen2-7B doesn't work properly in the example qwen

Open
#2,295 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

the Qwen/Qwen2-1.5B can work correct in the example, but Qwen/Qwen2-7B can't.

`(base) lyn@A100DEV:~/workspace/candle/candle-examples$ cargo run --release --features cuda --example qwen -- --model 2-7b --prompt "Hello\n"
Finished release [optimized] target(s) in 0.17s
Running `/disk/lyn/workspace/candle/target/release/examples/qwen --model 2-7b --prompt 'Hello\n'`
avx: true, neon: false, simd128: false, f16c: true
temp: 0.00 repeat-penalty: 1.10 repeat-last-n: 64
retrieved the files in 17.702388ms
loaded the model in 2.506053906s
Hello\n规范、、、、、、、 Orient Vall gu gap FourierbrtooltooltoolCharlesuketooltoolaurustooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltooltoolaurustoolrm哪些`

The model output is obviously incorrect.

the version:
commit 242e006bbb26ff12581b3c04bfd069996fe1f6bb (HEAD -> main, origin/main, origin/HEAD)
Author: Jeroen Vlek
Date: Mon Jun 24 19:12:52 2024 +0200

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `qwen` example in `candle-examples` and reproduce the reported command using the CUDA release build. Compare its 7B behavior with the working 1.5B model and trace the model-loading or generation path; done means the 7B example produces coherent output rather than the shown repeated tokens.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.