microsoft / microsoft/onnxruntime

[Mobile] [QNN EP GPU] Noisy ConvTranspose output differs completely from CPU output with same input

Open
#32,506 3 comments 0 reactions 0 assignees View on GitHub
ep:QNN platform:mobile
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Major update

This is most definitely **a problem with QNN rather than Ort**. I compiled [convstranspose_identity.tar.gz](https://github.com/user-attachments/files/32063034/convstranspose_identity.tar.gz) for use with QNN API using :
```bash
qnn-onnx-converter --input_network /path/to/convtranspose_1d_identity.onnxonnx --input_dim input 1,4,2 --output_path convtranspose_identity.cpp
qnn-model-lib-generator -c convtranspose_identity.cpp -b convtranspose_identity.bin -t aarch64-android -o convtranspose_identity_lib
```
Here is a complete usable and compilable program and scripts:
[reproduceable_qnn_inference.tar.gz](https://github.com/user-attachments/files/32096504/reproduceable_qnn_inference.tar.gz)

To use, change some paths in `build.sh` to match qairt paths and android ndk path and run `./build.sh`.
Push the `libconvtranspose_identity.so`, the built executable and Qnn's libs (`libQnnGpu.so`, `libQnnCpu.so`, `libQnnSystems.so`, ...) on the target, and run with:
```bash
./qnn_gpu_inference path/to/libQnn[Cpu,Gpu].so path/to/libconvtranspose_identity.so
```
On the target, when running on CPU, the output is:
```
[QNN] Tensors ready: 1 input(s), 1 output(s)
[QNN] Inference done. Output[0] values:
1 2 3 4 0 0 0 0 5 6 7 8 0 0 0 0 0 0 0 0 0 0 0 0
```

whereas GPU output is:
```
[QNN] Tensors ready: 1 input(s), 1 output(s)
[QNN] Inference done. Output[0] values:
1 2 3 4 5 6 7 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
```

### Describe the issue

This issue was encountered while testing inference of a Conv-TasNet model.

Given a model composed of a single `ConvTranspose` with the following attributes:
dilations: 1 group: 1 output_padding: 0 pads: 0,0 strides: 8 weight shape: 512x1x40

Its execution on GPU will produce noisy/erratic output that is completely different from the output it would produce on CPU, given the same input.

Here are some metrics' values comparing GPU vs CPU output for a same input:

Image

and a visual plot of the model output where clearly, gpu output and cpu output don't match:

Image

The phone used is Samsung S23 equipped with the Snapdragon 8 Gen 2 SM8550.

### To reproduce

With the provided minimal model : [minimal_convtranspose.tar.gz](https://github.com/user-attachments/files/32012440/minimal_convtranspose.tar.gz) and the following code which should be executed as `./program [cpu, gpu]`, one can easily obtain and compare the model's output on gpu and cpu:
```cpp
#include

#include
#include
#include
#include
#include

template
struct OrtTensorBuffer {
OrtTensorBuffer(const Ort::MemoryInfo& mem_info,
const std::array&& tensor_shape) :
shape{tensor_shape},
buffer_memory(std::accumulate(tensor_shape.begin(),
tensor_shape.end(),
int64_t{1},
std::multiplies{}),
0.f),
tensor{} {
if constexpr (std::same_as) {
tensor =
Ort::Value::CreateTensor(mem_info,
buffer_memory.data(),
buffer_memory.size() * sizeof(float),
shape.data(),
shape.size(),
ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT);
} else if constexpr (std::same_as) {
tensor = Ort::Value::CreateTensor(
mem_info,
buffer_memory.data(),
buffer_memory.size() * sizeof(uint16_t),
shape.data(),
shape.size(),
ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT16);
}
}

std::array shape;
std::vector buffer_memory;
Ort::Value tensor;
};

constexpr size_t upto = 20;
static constexpr size_t in_shape_size = 2;
static constexpr size_t out_shape_size = 3;
#define IN_SHAPE {2, 512, 2002}
#define OUT_SHAPE {2, 1, 16048}

int main(int argc, char** argv) {
Ort::Env env(ORT_LOGGING_LEVEL_VERBOSE, "convtranspose_debug\n");
Ort::SessionOptions session_options;
std::unordered_map qnn_options;
qnn_options["backend_type"] = argv[1];
session_options.AppendExecutionProvider("QNN", qnn_options);
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(
GraphOptimizationLevel::ORT_ENABLE_ALL);
session_options.SetLogSeverityLevel(0);
session_options.SetLogId("ort_session");
Ort::Session session =
Ort::Session(env, "convtranspose.onnx", session_options);

Ort::MemoryInfo memory_info =
Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
OrtTensorBuffer input_dummy{memory_info, IN_SHAPE};
OrtTensorBuffer output{memory_info, OUT_SHAPE};

// input is a sequence of random values

std::mt19937 rng(42);
std::uniform_real_distribution dist(-1.0f, 1.0f);
std::generate(input_dummy.buffer_memory.begin(),
input_dummy.buffer_memory.end(),
[&] { return dist(rng); });

Ort::AllocatorWithDefaultOptions allocator;
auto input_name_alloc = session.GetInputNameAllocated(0, allocator);
auto output_name_alloc = session.GetOutputNameAllocated(0, allocator);
const char* input_names[] = {input_name_alloc.get()};
const char* output_names[] = {output_name_alloc.get()};

Ort::IoBinding bindings{session};
bindings.ClearBoundInputs();
bindings.BindInput(input_names[0], input_dummy.tensor);
bindings.ClearBoundOutputs();
bindings.BindOutput(output_names[0], output.tensor);
session.Run(Ort::RunOptions{nullptr}, bindings);
for (const auto e : output.buffer_memory) {
std::cout << e << ",";
}
std::cout << std::endl << "Job done" << std::endl;

return 0;
}
```

### Urgency

Pretty urgent, this is part of a research project.

### Platform

Android

### OS Version

16

### ONNX Runtime Installation

Built from Source

### Compiler Version (if 'Built from Source')

Android clang version 21.0.0

### Package Name (if 'Released Package')

None

### ONNX Runtime Version or Commit ID

3f93e1f4

### ONNX Runtime API

C++/C

### Architecture

X64

### Execution Provider

Other / Unknown

### Execution Provider Library Version

QNN EP

Contributor guide

Open the contributing guide

Research direction

Start with minimal_convtranspose.tar.gz and the provided C++ program, running it with cpu and gpu to confirm the differing ConvTranspose outputs. Compare this with reproduceable_qnn_inference.tar.gz and the QNN commands, then determine whether the discrepancy is in ONNX Runtime's QNN EP or QNN itself. Done means the fault is isolated or CPU and GPU results are shown to match.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, cpp
Domain
machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.