[Bug]After converting sensevoice's onnx to trt via trtexec, an error is reported
Open
@asfiyab-nvidia is already working on this.
Since Jan 2, 2025.
triaged
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Currently, sensevoice's trt engine can be successfully converted through trtexec, but when running the benchmark infer, an error message is displayed as shown below:
ORT can be used to successfully predict the corresponding ONNX. The code is as follows, indicating that ONNX is fine
import onnxruntime
import torch
option = onnxruntime.SessionOptions()
option.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
option.intra_op_num_threads = 1
providers = [
"CUDAExecutionProvider"
if torch.cuda.is_available() else "CPUExecutionProvider"
]
model = onnxruntime.InferenceSession(
"model_sensevoice.onnx",
sess_options=option, providers=providers)
batch_size = 4
feats_length = 256
speech = torch.randn(batch_size, feats_length, 560).cuda()
speech_lengths = torch.tensor([6, 30, 31, feats_length], dtype=torch.int32).cuda()
language = torch.tensor([0, 0, 0, 0], dtype=torch.int32).cuda()
textnorm = torch.tensor([15, 15, 15, 15], dtype=torch.int32).cuda()
ort_inputs = {
'speech': speech.cpu().numpy(),
'speech_lengths': speech_lengths.cpu().numpy(),
'language': language.cpu().numpy(),
'textnorm': textnorm.cpu().numpy(),
}
output = model.run(None, ort_inputs)[0]
print("output:", output, output.shape)
trtexec convert:
trtexec \
--onnx=model_sensevoice.onnx \
--saveEngine=engine_fp16.plan \
--minShapes=speech:1x128x560,speech_lengths:1,language:1,textnorm:1 \
--optShapes=speech:4x256x560,speech_lengths:4,language:4,textnorm:4 \
--maxShapes=speech:8x512x560,speech_lengths:8,language:8,textnorm:8 \
--fp16 \
--builderOptimizationLevel=3 \
--memPoolSize=workspace:4096 \
--verbose
TRT version:TensorRT-10.7.0.23
ONNX version: 1.17.0
So what is the specific reason? Thank you~
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.