pytorch / pytorch/executorch

Follow the guide of qnn Android demo to run until coredump occurs during qnn model quantization

Open
#7,437 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

module: qnn partner: qualcomm triaged
Dominant language
Python
Stars
5k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
581

Description

πŸ› Describe the bug

https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/LlamaDemo/docs/delegates/qualcomm_README.md
I followed this demo to start quantifying the model. The beginning was smooth, but the flatbuffer assertion would occur at the end. My model was llama2-7b and quantized by a16w4, which could not generate the final pte for qnn. I've also tried --num_sharding 4, but other problems pop up

PTQ# 4 bits weight only quantize
python -m examples.models.llama.export_llama --checkpoint "${MODEL_DIR}/consolidated.00.pth" -p "${MODEL_DIR}/params.json" -kv --disable_dynamic_shape --qnn --pt2e_quantize qnn_16a4w -d fp32 --metadata '{"get_bos_id":128000, "get_eos_ids":[128009, 128001]}' --output_name="test.pte”
It starts out fine, but ends up with the following problem
β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”

INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_convolution_default_224, aten.convolution.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_permute_copy_default_3493, aten.permute_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_view_copy_default_577, aten.view_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: quantized_decomposed_dequantize_per_tensor_tensor, quantized_decomposed.dequantize_per_tensor.tensor
python: /002data/andrea/executorch/qnn/executorch/third-party/flatbuffers/include/flatbuffers/vector_downward.h:146: size_t flatbuffers::vector_downward::ensure_space(size_t) [with SizeT = unsigned int; size_t = long unsigned int]: Assertion `size() < max_size_' failed.
Aborted (core dumped)

Versions

source code build v0.5 follow the android llama demo for qualcomm

cc @cccclai @winskuo-quic @shewu-quic

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up β€” it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/demo-apps/android/LlamaDemo/docs/delegates/qualcomm_README.md and reproduce the reported llama2-7b quantization command using the qnn_16a4w option. Then trace the final export around the flatbuffers vector_downward assertion and the QNN preprocessing log; done means the model produces a final .pte without the coredump, or the failure is narrowed to a documented reproducible cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, python, pytorch
Domain
machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.