microsoft / microsoft/onnxruntime

Inputs with UINT8 should fallback to CUDA instead of failing to compile with TRT execution provider

Open
#5,785 17 comments 0 reactions 1 assignee View on GitHub

@stevenlix is already working on this.

Since Dec 14, 2020.

ep:CUDA ep:TensorRT feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

Models with uint8 inputs are not supported by TRT execution provider. This is fine, but the node should fallback to CUDA, instead of failing to to the conversion in onnxruntime.

**System information**
- ONNX Runtime version (you are using):
1.5.2

**Describe the solution you'd like**
In my case the first operation in the model is a cast to fp32 anyway, according to the onnx profiler this costs **0.01ms** on CUDA.
Compared to doing this cast on a 512x616 grayscale image on the caller side before:
**Python**(numpy astype): **0.8ms**
**C#** (static array definition and copy with BlockCopy): **1ms**

Also int8 is not really an alternative, images/data that is received, either overflows are needs to be shifted, which also costs 0.65ms in python. Furthermore it needs retraining/further changes on the model (correct me if there is a smarter way).

Hence, just executing this operatoin in CUDA should be cheaper, either by beeing able to mark some operations to force them to run on a defined executor, or just hardcode it for this kind of casts.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Tested on Nvidia T4/Quadro Intel and AMD CPUs

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.