NVIDIA / NVIDIA/TensorRT

How to use calibrated cache file to generate int8 engine in c++

Open
#3,881 1 comment 0 reactions 1 assignee View on GitHub

@brb-nv is already working on this.

Since May 24, 2024.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

I have successfully generated calibrated dataset.cache file of my dataset using polygraphy. I want to load the generated calibrated cache file and create int8 engine using c++.

Function I'm using to convert onnx model into tensorrt engine file

std::unique_ptr<nvinfer1::ICudaEngine> createCudaEngine(const std::string &onnxFileName, nvinfer1::ILogger &logger, int batchSize, ENGINE_TYPE& type=FP32)
 {
    std::unique_ptr<IBuilder> builder = createInferBuilder(logger);
    std::unique_ptr<INetworkDefinition> network = builder->createNetworkV2(1U << (unsigned) NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);

    std::unique_ptr<nvonnxparser::IParser> parser = nvonnxparser::createParser(*network, logger);

    if (!parser->parseFromFile(onnxFileName.c_str(), static_cast<int>(ILogger::Severity::kINFO)))
    throw std::runtime_error("ERROR: could not parse ONNX model " + onnxFileName + " !");
    std::unique_ptr<IOptimizationProfile> profile = builder->createOptimizationProfile();
    profile->setDimensions("input", OptProfileSelector::kMIN, Dims2{batchSize, 3});
    profile->setDimensions("input", OptProfileSelector::kMAX, Dims2{batchSize, 3});
    profile->setDimensions("input", OptProfileSelector::kOPT, Dims2{batchSize, 3});

    IBuilderConfig* config = builder->createBuilderConfig();

    config->setMaxWorkspaceSize(64*1024*1024);
    config->addOptimizationProfile(profile);

    if (INT8) 
    {   
       // how to load and use calibrated cache file here 
        config->setFlag(BuilderFlag::kINT8);
    }
    return builder->buildEngineWithConfig(*network, *config);
 }

@zerollzeng Please help

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.