How to use calibrated cache file to generate int8 engine in c++
Open
@brb-nv is already working on this.
Since May 24, 2024.
triaged
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
I have successfully generated calibrated dataset.cache file of my dataset using polygraphy. I want to load the generated calibrated cache file and create int8 engine using c++.
Function I'm using to convert onnx model into tensorrt engine file
std::unique_ptr<nvinfer1::ICudaEngine> createCudaEngine(const std::string &onnxFileName, nvinfer1::ILogger &logger, int batchSize, ENGINE_TYPE& type=FP32)
{
std::unique_ptr<IBuilder> builder = createInferBuilder(logger);
std::unique_ptr<INetworkDefinition> network = builder->createNetworkV2(1U << (unsigned) NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
std::unique_ptr<nvonnxparser::IParser> parser = nvonnxparser::createParser(*network, logger);
if (!parser->parseFromFile(onnxFileName.c_str(), static_cast<int>(ILogger::Severity::kINFO)))
throw std::runtime_error("ERROR: could not parse ONNX model " + onnxFileName + " !");
std::unique_ptr<IOptimizationProfile> profile = builder->createOptimizationProfile();
profile->setDimensions("input", OptProfileSelector::kMIN, Dims2{batchSize, 3});
profile->setDimensions("input", OptProfileSelector::kMAX, Dims2{batchSize, 3});
profile->setDimensions("input", OptProfileSelector::kOPT, Dims2{batchSize, 3});
IBuilderConfig* config = builder->createBuilderConfig();
config->setMaxWorkspaceSize(64*1024*1024);
config->addOptimizationProfile(profile);
if (INT8)
{
// how to load and use calibrated cache file here
config->setFlag(BuilderFlag::kINT8);
}
return builder->buildEngineWithConfig(*network, *config);
}
@zerollzeng Please help
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.