pytorch / pytorch/executorch

qwen2.5 0.5b inference with llama.py is normal, but qwen3 0.6b repeats

Open
#14,402 28 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

module: qnn partner: qualcomm
Dominant language
Python
Stars
5k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
581

Description

Using llama.py to convert qwen2.5 0.5b produces correct responses. The conversion command is as follows:

python examples/qualcomm/oss_scripts/llama/llama.py \
  -b build-android \
  -s 4c576cc \
  -m SM8650 \
  --temperature 0 \
  --model_mode hybrid \
  --max_seq_len 1024 \
  --prefill_ar_len 128 \
  --decoder_model qwen2_5-0_5b \
  --prompt "I would like to learn python, could you teach me with a simple example?" \
  --tasks wikitext \
  --limit 1 \
  --compile_only

Then use the following command for inference:

python examples/qualcomm/oss_scripts/llama/llama.py \
  -b build-android \
  -s 4c576cc \
  -m SM8650 \
  --decoder_model qwen2_5-0_5b \
  --model_mode hybrid \
  --prefill_ar_len 128 \
  --max_seq_len 1024 \
  --prompt "Write a bubble sort in C++." \
  --pre_gen_pte /executorch/llama_qnn

The response is correct as follow:

Image

Using the same method, converting qwen3 0.6b with llama.py:

python examples/qualcomm/oss_scripts/llama/llama.py \
  -b build-android \
  -s 4c576cc \
  -m SM8650 \
  --temperature 0 \
  --model_mode hybrid \
  --max_seq_len 1024 \
  --prefill_ar_len 128 \
  --decoder_model qwen3-0_6b \
  --prompt "I would like to learn python, could you teach me with a simple example?" \
  --tasks wikitext \
  --limit 1 \
  --compile_only

Then perform inference using the following command:

python examples/qualcomm/oss_scripts/llama/llama.py \
  -b build-android \
  -s 4c576cc \
  -m SM8650 \
  --decoder_model qwen3-0_6b \
  --model_mode hybrid \
  --prefill_ar_len 128 \
  --max_seq_len 1024 \
  --prompt "Write a bubble sort in C++." \
  --pre_gen_pte /executorch/llama_qnn

The response is repetitive as follows:

Image

how to slove the problem? thanks !

cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/qualcomm/oss_scripts/llama/llama.py and reproduce the qwen2_5-0_5b and qwen3-0_6b conversion and inference commands on SM8650. Compare the generated responses and conversion settings; done means qwen3-0_6b produces a non-repetitive response for the supplied prompt.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, python
Domain
embedded-iot, machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.