NVIDIA / NVIDIA/TensorRT

Core dumped failure of TensorRT 8.6 when building FP16 model on GPU 2080ti

Open
#3,162 5 comments 0 reactions 1 assignee View on GitHub

@zerollzeng is already working on this.

Since Jul 29, 2023.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

Build the attached onnx model in fp16 precision using TensorRT 8.6 on 2080ti will cause the following error. However, everything works fine when build model in fp32 precision on 2080ti and it also works fine on 3090, no matter in fp16 or fp32.

image

Error message:
fusion_codegen.cpp:91: DCHECK(graph_shader && "compilation failed") failed.
terminate called after throwing an instance of 'MyelinCheckException'
what(): MyelinCheckException
Aborted (core dumped)

Update:
It seems that this is caused by the fusion of MatMul and Einsum op, by replacing the Einsum with Transpose and MatMul, everything works fine.
image

Environment

TensorRT Version: 8.6.1

NVIDIA GPU: 2080TI

NVIDIA Driver Version: 525

CUDA Version: CUDA in image nvcr.io/nvidia/tensorrt:23.06-py3

CUDNN Version: CUDNN in image nvcr.io/nvidia/tensorrt:23.06-py3

Operating System: OS in image nvcr.io/nvidia/tensorrt:23.06-py3

Python Version (if applicable):

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

Model link:

bug_fp16.zip
fine_fp16.zip

Steps To Reproduce

Commands or scripts:

trtexec --onnx=bug_fp16.onnx --saveEngine=./bug_fp16.plan --fp16

Have you tried the latest release?: YES

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): YES

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.