qwen2.5 0.5b inference with llama.py is normal, but qwen3 0.6b repeats
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
Using llama.py to convert qwen2.5 0.5b produces correct responses. The conversion command is as follows:
python examples/qualcomm/oss_scripts/llama/llama.py \
-b build-android \
-s 4c576cc \
-m SM8650 \
--temperature 0 \
--model_mode hybrid \
--max_seq_len 1024 \
--prefill_ar_len 128 \
--decoder_model qwen2_5-0_5b \
--prompt "I would like to learn python, could you teach me with a simple example?" \
--tasks wikitext \
--limit 1 \
--compile_only
Then use the following command for inference:
python examples/qualcomm/oss_scripts/llama/llama.py \
-b build-android \
-s 4c576cc \
-m SM8650 \
--decoder_model qwen2_5-0_5b \
--model_mode hybrid \
--prefill_ar_len 128 \
--max_seq_len 1024 \
--prompt "Write a bubble sort in C++." \
--pre_gen_pte /executorch/llama_qnn
The response is correct as follow:
Using the same method, converting qwen3 0.6b with llama.py:
python examples/qualcomm/oss_scripts/llama/llama.py \
-b build-android \
-s 4c576cc \
-m SM8650 \
--temperature 0 \
--model_mode hybrid \
--max_seq_len 1024 \
--prefill_ar_len 128 \
--decoder_model qwen3-0_6b \
--prompt "I would like to learn python, could you teach me with a simple example?" \
--tasks wikitext \
--limit 1 \
--compile_only
Then perform inference using the following command:
python examples/qualcomm/oss_scripts/llama/llama.py \
-b build-android \
-s 4c576cc \
-m SM8650 \
--decoder_model qwen3-0_6b \
--model_mode hybrid \
--prefill_ar_len 128 \
--max_seq_len 1024 \
--prompt "Write a bubble sort in C++." \
--pre_gen_pte /executorch/llama_qnn
The response is repetitive as follows:
how to slove the problem? thanks !
cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/qualcomm/oss_scripts/llama/llama.py and reproduce the qwen2_5-0_5b and qwen3-0_6b conversion and inference commands on SM8650. Compare the generated responses and conversion settings; done means qwen3-0_6b produces a non-repetitive response for the supplied prompt.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, python
- Domain
- embedded-iot, machine-learning, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100