QNN LLM Compilation Pipeline — Profiling & Optimization

TLDR

Compiling quantized LLMs for the QNN HTP backend takes 5 to 12+ hours end-to-end. This blocks iteration on model quality, quantization recipes, and on-device evaluation. The pipeline has three major bottlenecks:

1. Wasted autoregressive generation during calibration: After prefilling a short prompt into the KV cache, the calibration loop runs _generate to produce hundreds of tokens auto-regressively. These generated tokens are concat to the prompt and we run calibration with prompt + generated token, i don't think this improves quantization observer statistics since we use generated data by the model.

2. Slow per-iteration DECODE calibration: The DECODE model processes tokens one at a time (ar_len=1), requiring 1023 sequential forward passes per wikitext chunk. Each pass takes 2-5s on CPU, making task calibration alone ~90 min. This needs deep dive into the torchao/ pytorch/ QNN quantization to find any optimizations to improve the per iteration speed.

3. QNN SDK HTP compilation: qnn_manager.Compile (specifically GraphFinalize) runs the HTP compiler's optimization passes at level 3 (most aggressive). This takes 102-190 min depending on model size.

Baseline Profiling

Model: qwen2_5-1_5b (hybrid, max_seq_len=1024, prefill_ar_len=128)

python examples/qualcomm/oss_scripts/llama/llama.py -b build-android --compile_only -m SM8750 \
  --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 \
  --decoder_model qwen2_5-1_5b \
  --prompt "I would like to learn python, could you teach me with a simple example?" \
  --tasks wikitext --limit 1

Model: qwen3-1_7b (hybrid, max_seq_len=1024, prefill_ar_len=128)

Bottleneck Summary

Flame graph

Notes:

1. I couldn't run multiple runs per model due to long running nature and hence the percentages might have inherent overall and sub-category variance.
2. I haven't verified ppl on-device, to get fast feedback created the issue with data and approaches first, to check if there is missing context.

cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin @kimishpatel @jerryzh168