Core dumped failure of TensorRT 8.6 when building FP16 model on GPU 2080ti
@zerollzeng is already working on this.
Since Jul 29, 2023.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
Build the attached onnx model in fp16 precision using TensorRT 8.6 on 2080ti will cause the following error. However, everything works fine when build model in fp32 precision on 2080ti and it also works fine on 3090, no matter in fp16 or fp32.
Error message:
fusion_codegen.cpp:91: DCHECK(graph_shader && "compilation failed") failed.
terminate called after throwing an instance of 'MyelinCheckException'
what(): MyelinCheckException
Aborted (core dumped)
Update:
It seems that this is caused by the fusion of MatMul and Einsum op, by replacing the Einsum with Transpose and MatMul, everything works fine.
Environment
TensorRT Version: 8.6.1
NVIDIA GPU: 2080TI
NVIDIA Driver Version: 525
CUDA Version: CUDA in image nvcr.io/nvidia/tensorrt:23.06-py3
CUDNN Version: CUDNN in image nvcr.io/nvidia/tensorrt:23.06-py3
Operating System: OS in image nvcr.io/nvidia/tensorrt:23.06-py3
Python Version (if applicable):
Tensorflow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if so, version):
Relevant Files
Model link:
Steps To Reproduce
Commands or scripts:
trtexec --onnx=bug_fp16.onnx --saveEngine=./bug_fp16.plan --fp16
Have you tried the latest release?: YES
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): YES
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.