In trt10.0.1, these two APIs: setPrecision and setOutputType do not work
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
We have a model that overflows when using fp16, so we use layer-precision to limit it and let some layers use fp32. It worked in version 8.6 and we could infer normal results. But after upgrading to 10.0.1, we found that the model output overflowed. Using polygraphy, we found that nan was already generated at the first overflow location (Is setprecison and setoutputType invalid?)
Environment
TensorRT Version:
10.0.1
NVIDIA GPU:
3090 & 3080
NVIDIA Driver Version:
550
CUDA Version:
cuda-12.2
Steps To Reproduce
my code is like this:
for (int32_t layerIdx = 0; layerIdx < network.getNbLayers(); ++layerIdx) {
auto *layer = network.getLayer(layerIdx);
auto const layerName = layer->getName();
nvinfer1::DataType dataType;
if (matchLayerPrecision(layerPrecisions, layerName, &dataType)) { // Function to determine whether to limit the precision
layer->setPrecision(dataType);
int32_t layerOutNb = layer->getNbOutputs();
for (int32_t outputIdx = 0; outputIdx < layerOutNb; outputIdx++) {
layer->setOutputType(outputIdx, dataType);
}}}
By the way, I have already set kOBEY_PRECISION_CONSTRAINTS
env.config_->setFlag(nvinfer1::BuilderFlag::kOBEY_PRECISION_CONSTRAINTS);
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided setPrecision and setOutputType loop in TensorRT 10.0.1 with the listed GPU, driver, and CUDA environment, then compare behavior with version 8.6. Investigate the first layer producing NaN and verify whether kOBEY_PRECISION_CONSTRAINTS is honored. Done means a minimal reproducible case and a confirmed explanation or fix for the precision constraint regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100