pytorch / pytorch/vision

Replicate PyTorch transforms like transforms.ToTensor() and transforms.Normalize() in opencv c++

Open
#6,183 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

Issue description

In my development code, I have pytorch transforms on my test dataset (torchvision.transforms), I have exported the model into ONNX and I want to replicate these transforms with OpenCV c++. but, when i make predictions using pytorch (with the exported ONNX model), I get different result than when making predictions with OpenCV c++

PyTorch transforms


test_transforms = transforms.Compose([
                           transforms.Resize(pretrained_size),
                           transforms.CenterCrop(pretrained_size),
                           transforms.ToTensor(),
                           transforms.Normalize(mean = pretrained_means, 
                                                std = pretrained_stds)
                       ])

  • Output of Inference with PyTorch:
pytorch image type: torch.float32
detected class: cross-over
confidence: 0.8553193807601929
output probabilities of softmax layer: [[2.9558592e-05 8.5531938e-01 6.2924426e-04 1.1440608e-02 5.4786936e-04
  4.9833752e-02 2.7969838e-04 8.1919849e-02]]

My Inference with OpenCV c++

    py::dict VClassdict;

    // Convert input Image from Python Code to Numpy Array
    Mat frame = nparray_to_mat(img);

    // Color Conversion to RGB
    Mat Img;
    //cvtColor(frame, Img, COLOR_BGR2RGB);

    // Resize Image to ResNet Input size
    cv::resize(frame, Img, Size(ClassinpWidth, ClassinpWidth));

    Mat normImg;
    Img.convertTo(normImg, CV_32FC3, 1.f / 255);
    Scalar mean;
    Scalar std;
    cv::meanStdDev(normImg, mean, std);

    mean[0] *= 255.0;
    mean[1] *= 255.0;
    mean[2] *= 255.0;

    double scale_factor = 0.003921569;  // equivalent to 1/255
   // Scalar mean = Scalar(117.8865, 128.52, 137.0115); // each channel mean is multiplied by 255.0
    //Scalar std = Scalar(0.2275, 0.2110, 0.2140);
    bool swapRB = true;
    bool crop = false;

    Mat blob;
    cv::dnn::blobFromImage(Img, blob, scale_factor, Size(ClassinpWidth, ClassinpWidth), mean, swapRB, crop);

    if (std.val[0] != 0.0 && std.val[1] != 0.0 && std.val[2] != 0.0)
    {
        // Divide blob by std.
        divide(blob, std, blob);
    }

    VClassResNet_Net.setInput(blob);

    // predict
    Mat prob = VClassResNet_Net.forward();

    cout << "output probabilities of softmax layer: " << prob;
    cout << endl;

    // extract prediction with highest confidence
    Point classIdPoint;
    double confidence;
    minMaxLoc(prob.reshape(1, 1), 0, &confidence, 0, &classIdPoint);
    int classId = classIdPoint.x;

    // Setup Dict returned to Python code for detections
    VClassdict["ObjectClass"] = VClassResNet_classes[classId].c_str(); // Detected Class
    VClassdict["Confidence"] = confidence; // Confedence Level
  • Output of OpenCV c++ Inference:
detected class: cross-over
confidence: 0.9028045535087585
output probabilities of softmax layer: [2.5416075e-05, 0.90280455, 0.0031091773, 0.0042484328, 0.00012638989, 0.05069441, 9.7391217e-05, 0.038894258]

I've followed the OpenCV documentation : https://docs.opencv.org/4.x/dd/d55/pytorch_cls_c_tutorial_dnn_conversion.html

System Info

  • PyTorch version: 1.10.2+cu113
  • OpenCV version: 4.5.1
  • Python version: 3.9.7
  • OS: Windows

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository file or test is identified. Start by comparing the torchvision preprocessing sequence with the OpenCV C++ code, using the linked OpenCV DNN conversion documentation and the reported inference outputs; done means determining why equivalent inputs produce different probabilities.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.