NVIDIA / NVIDIA/TensorRT

How does folding constant help improve the trt engine performance

Open
#2,937 1 comment 0 reactions 1 assignee View on GitHub

@BowenFu is already working on this.

Since May 7, 2023.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Hi, tensorrt team! I found if I fold constants when exporting the onnx, the trt engine built based on the onnx would be faster when inferring. Could I get an explanation from you?
Besides, in order to refit an engine, we build a mapping relationship between torch model and tensorrt engine without folding constant because folding constant would break the relationship between them (module names will be different). Do you have any approaches to build a relationship between torch model and tensorrt engine which constants were folded, or if not folding constant is necessary, do you have a method to decrease the inferring time?
Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.