NVIDIA / NVIDIA/TensorRT

[Bug]After converting sensevoice's onnx to trt via trtexec, an error is reported

Open
#4,307 6 comments 0 reactions 1 assignee View on GitHub

@asfiyab-nvidia is already working on this.

Since Jan 2, 2025.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Currently, sensevoice's trt engine can be successfully converted through trtexec, but when running the benchmark infer, an error message is displayed as shown below:
Image

ORT can be used to successfully predict the corresponding ONNX. The code is as follows, indicating that ONNX is fine

import onnxruntime
import torch

option = onnxruntime.SessionOptions()
option.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
option.intra_op_num_threads = 1
providers = [
    "CUDAExecutionProvider" 
    if torch.cuda.is_available() else "CPUExecutionProvider"
]
model = onnxruntime.InferenceSession(
    "model_sensevoice.onnx",
    sess_options=option, providers=providers)

batch_size = 4
feats_length = 256
speech = torch.randn(batch_size, feats_length, 560).cuda()
speech_lengths = torch.tensor([6, 30, 31, feats_length], dtype=torch.int32).cuda()
language = torch.tensor([0, 0, 0, 0], dtype=torch.int32).cuda()
textnorm = torch.tensor([15, 15, 15, 15], dtype=torch.int32).cuda()

ort_inputs = {
    'speech': speech.cpu().numpy(),
    'speech_lengths': speech_lengths.cpu().numpy(),
    'language': language.cpu().numpy(),
    'textnorm': textnorm.cpu().numpy(),
}
output = model.run(None, ort_inputs)[0]
print("output:", output, output.shape)

trtexec convert:

trtexec \
    --onnx=model_sensevoice.onnx \
    --saveEngine=engine_fp16.plan \
    --minShapes=speech:1x128x560,speech_lengths:1,language:1,textnorm:1 \
    --optShapes=speech:4x256x560,speech_lengths:4,language:4,textnorm:4 \
    --maxShapes=speech:8x512x560,speech_lengths:8,language:8,textnorm:8 \
    --fp16 \
    --builderOptimizationLevel=3 \
    --memPoolSize=workspace:4096 \
    --verbose

TRT version:TensorRT-10.7.0.23
ONNX version: 1.17.0

So what is the specific reason? Thank you~

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.