microsoft / microsoft/onnxruntime
[Mobile] [QNN EP GPU] Noisy ConvTranspose output differs completely from CPU output with same input
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Major update
This is most definitely **a problem with QNN rather than Ort**. I compiled [convstranspose_identity.tar.gz](https://github.com/user-attachments/files/32063034/convstranspose_identity.tar.gz) for use with QNN API using :
```bash
qnn-onnx-converter --input_network /path/to/convtranspose_1d_identity.onnxonnx --input_dim input 1,4,2 --output_path convtranspose_identity.cpp
qnn-model-lib-generator -c convtranspose_identity.cpp -b convtranspose_identity.bin -t aarch64-android -o convtranspose_identity_lib
```
Here is a complete usable and compilable program and scripts:
[reproduceable_qnn_inference.tar.gz](https://github.com/user-attachments/files/32096504/reproduceable_qnn_inference.tar.gz)
To use, change some paths in `build.sh` to match qairt paths and android ndk path and run `./build.sh`.
Push the `libconvtranspose_identity.so`, the built executable and Qnn's libs (`libQnnGpu.so`, `libQnnCpu.so`, `libQnnSystems.so`, ...) on the target, and run with:
```bash
./qnn_gpu_inference path/to/libQnn[Cpu,Gpu].so path/to/libconvtranspose_identity.so
```
On the target, when running on CPU, the output is:
```
[QNN] Tensors ready: 1 input(s), 1 output(s)
[QNN] Inference done. Output[0] values:
1 2 3 4 0 0 0 0 5 6 7 8 0 0 0 0 0 0 0 0 0 0 0 0
```
whereas GPU output is:
```
[QNN] Tensors ready: 1 input(s), 1 output(s)
[QNN] Inference done. Output[0] values:
1 2 3 4 5 6 7 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
```
### Describe the issue
This issue was encountered while testing inference of a Conv-TasNet model.
Given a model composed of a single `ConvTranspose` with the following attributes:
dilations: 1 group: 1 output_padding: 0 pads: 0,0 strides: 8 weight shape: 512x1x40
Its execution on GPU will produce noisy/erratic output that is completely different from the output it would produce on CPU, given the same input.
Here are some metrics' values comparing GPU vs CPU output for a same input:
and a visual plot of the model output where clearly, gpu output and cpu output don't match:
The phone used is Samsung S23 equipped with the Snapdragon 8 Gen 2 SM8550.
### To reproduce
With the provided minimal model : [minimal_convtranspose.tar.gz](https://github.com/user-attachments/files/32012440/minimal_convtranspose.tar.gz) and the following code which should be executed as `./program [cpu, gpu]`, one can easily obtain and compare the model's output on gpu and cpu:
```cpp
#include
#include
#include
#include
#include
#include
template
struct OrtTensorBuffer {
OrtTensorBuffer(const Ort::MemoryInfo& mem_info,
const std::array&& tensor_shape) :
shape{tensor_shape},
buffer_memory(std::accumulate(tensor_shape.begin(),
tensor_shape.end(),
int64_t{1},
std::multiplies{}),
0.f),
tensor{} {
if constexpr (std::same_as) {
tensor =
Ort::Value::CreateTensor(mem_info,
buffer_memory.data(),
buffer_memory.size() * sizeof(float),
shape.data(),
shape.size(),
ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT);
} else if constexpr (std::same_as) {
tensor = Ort::Value::CreateTensor(
mem_info,
buffer_memory.data(),
buffer_memory.size() * sizeof(uint16_t),
shape.data(),
shape.size(),
ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT16);
}
}
std::array shape;
std::vector buffer_memory;
Ort::Value tensor;
};
constexpr size_t upto = 20;
static constexpr size_t in_shape_size = 2;
static constexpr size_t out_shape_size = 3;
#define IN_SHAPE {2, 512, 2002}
#define OUT_SHAPE {2, 1, 16048}
int main(int argc, char** argv) {
Ort::Env env(ORT_LOGGING_LEVEL_VERBOSE, "convtranspose_debug\n");
Ort::SessionOptions session_options;
std::unordered_map qnn_options;
qnn_options["backend_type"] = argv[1];
session_options.AppendExecutionProvider("QNN", qnn_options);
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(
GraphOptimizationLevel::ORT_ENABLE_ALL);
session_options.SetLogSeverityLevel(0);
session_options.SetLogId("ort_session");
Ort::Session session =
Ort::Session(env, "convtranspose.onnx", session_options);
Ort::MemoryInfo memory_info =
Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
OrtTensorBuffer input_dummy{memory_info, IN_SHAPE};
OrtTensorBuffer output{memory_info, OUT_SHAPE};
// input is a sequence of random values
std::mt19937 rng(42);
std::uniform_real_distribution dist(-1.0f, 1.0f);
std::generate(input_dummy.buffer_memory.begin(),
input_dummy.buffer_memory.end(),
[&] { return dist(rng); });
Ort::AllocatorWithDefaultOptions allocator;
auto input_name_alloc = session.GetInputNameAllocated(0, allocator);
auto output_name_alloc = session.GetOutputNameAllocated(0, allocator);
const char* input_names[] = {input_name_alloc.get()};
const char* output_names[] = {output_name_alloc.get()};
Ort::IoBinding bindings{session};
bindings.ClearBoundInputs();
bindings.BindInput(input_names[0], input_dummy.tensor);
bindings.ClearBoundOutputs();
bindings.BindOutput(output_names[0], output.tensor);
session.Run(Ort::RunOptions{nullptr}, bindings);
for (const auto e : output.buffer_memory) {
std::cout << e << ",";
}
std::cout << std::endl << "Job done" << std::endl;
return 0;
}
```
### Urgency
Pretty urgent, this is part of a research project.
### Platform
Android
### OS Version
16
### ONNX Runtime Installation
Built from Source
### Compiler Version (if 'Built from Source')
Android clang version 21.0.0
### Package Name (if 'Released Package')
None
### ONNX Runtime Version or Commit ID
3f93e1f4
### ONNX Runtime API
C++/C
### Architecture
X64
### Execution Provider
Other / Unknown
### Execution Provider Library Version
QNN EP
Contributor guide
Research direction
Start with minimal_convtranspose.tar.gz and the provided C++ program, running it with cpu and gpu to confirm the differing ConvTranspose outputs. Compare this with reproduceable_qnn_inference.tar.gz and the QNN commands, then determine whether the discrepancy is in ONNX Runtime's QNN EP or QNN itself. Done means the fault is isolated or CPU and GPU results are shown to match.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, cpp
- Domain
- machine-learning, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100