Follow the guide of qnn Android demo to run until coredump occurs during qnn model quantization
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
π Describe the bug
https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/LlamaDemo/docs/delegates/qualcomm_README.md
I followed this demo to start quantifying the model. The beginning was smooth, but the flatbuffer assertion would occur at the end. My model was llama2-7b and quantized by a16w4, which could not generate the final pte for qnn. I've also tried --num_sharding 4, but other problems pop up
PTQ# 4 bits weight only quantize
python -m examples.models.llama.export_llama --checkpoint "${MODEL_DIR}/consolidated.00.pth" -p "${MODEL_DIR}/params.json" -kv --disable_dynamic_shape --qnn --pt2e_quantize qnn_16a4w -d fp32 --metadata '{"get_bos_id":128000, "get_eos_ids":[128009, 128001]}' --output_name="test.pteβ
It starts out fine, but ends up with the following problem
ββββββββββββββββββββββββββββββββββββββββββββββββββββ
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_convolution_default_224, aten.convolution.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_permute_copy_default_3493, aten.permute_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: aten_view_copy_default_577, aten.view_copy.default
INFO:executorch.backends.qualcomm.qnn_preprocess:Visiting: quantized_decomposed_dequantize_per_tensor_tensor, quantized_decomposed.dequantize_per_tensor.tensor
python: /002data/andrea/executorch/qnn/executorch/third-party/flatbuffers/include/flatbuffers/vector_downward.h:146: size_t flatbuffers::vector_downward::ensure_space(size_t) [with SizeT = unsigned int; size_t = long unsigned int]: Assertion `size() < max_size_' failed.
Aborted (core dumped)
Versions
source code build v0.5 follow the android llama demo for qualcomm
cc @cccclai @winskuo-quic @shewu-quic
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up β it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/demo-apps/android/LlamaDemo/docs/delegates/qualcomm_README.md and reproduce the reported llama2-7b quantization command using the qnn_16a4w option. Then trace the final export around the flatbuffers vector_downward assertion and the QNN preprocessing log; done means the model produces a final .pte without the coredump, or the failure is narrowed to a documented reproducible cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, python, pytorch
- Domain
- machine-learning, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100