pytorch / pytorch/vision

Cannot obtain the accuracy stated in the doc for inception_v3 pretrained on Imagenet

Open
#6,066 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

Hi.
I'm trying to evaluate inception_v3 pretrained model from the hub on Imagenet (ILSVRC 2012) test set. I use the following evaluation code:

def compute_accuracy(self):
        num_correct = 0
        num_images = 0

        _IMAGE_MEAN_VALUE = [0.485, 0.456, 0.406]
        _IMAGE_STD_VALUE = [0.229, 0.224, 0.225]
        imgnet_loader = torch.utils.data.DataLoader(
            ImageFolder('/home/amin/dataset/ILSVRC/val', 
            transforms.Compose([
                transforms.Resize(299),
                transforms.CenterCrop(299),
                transforms.ToTensor(),
                transforms.Normalize(mean=_IMAGE_MEAN_VALUE, std=_IMAGE_STD_VALUE),
            ])),
            batch_size=16, shuffle=True,
            num_workers=8, pin_memory=True)
        self.model = torch.hub.load('pytorch/vision:v0.10.0', 'inception_v3', pretrained=True).cuda()
        self.model.eval()

        for i, (images, targets) in \
            enumerate(tqdm(imgnet_loader,  desc="Compute Accuracy", total=len(imgnet_loader))):
            images = images.cuda() if torch.cuda.is_available() else images.cpu()
            targets = targets.cuda() if torch.cuda.is_available() else targets.cpu()
            output_dict = self.model(images)
            pred=output_dict.argmax(dim=1)
            
            num_correct += (pred == targets).sum().item()
            num_images += images.size(0)

        classification_acc = num_correct / float(num_images) * 100
        return classification_acc

However, the accuracy I get is 77.216 while it should be 77.45 according to this page. I figured that the model has a transform_input as preprocess in itself. So if we are doing normalization beforehand (as suggested in the example code), we should set transform_input to false. So if I add self.model.transform_input = False, I get 77.472, Which is closer to the expected value but not exactly the same.

Assuming that the issue isn't from my code, I also found the thread on std, mean values (#1439). So I tested some of the values suggested there as well, and got these results:

mean std model's transform_input Accuracy
[0.485, 0.456, 0.406] [0.229, 0.224, 0.225] disabled 77.472
[0.485, 0.456, 0.406] [0.229, 0.224, 0.225] enabled 77.216
[0.4803, 0.4569, 0.4083] [0.2806, 0.2736, 0.2877] disabled 77.448
[0.4803, 0.4569, 0.4083] [0.2806, 0.2736, 0.2877] enabled 76.986
[0.4845, 0.4541, 0.4025] [0.2724, 0.2637, 0.2761] disabled 77.456
[0.4845, 0.4541, 0.4025] [0.2724, 0.2637, 0.2761] enabled 77.03
[0.4701, 0.4340, 0.3832] [0.2845, 0.2733, 0.2805] disabled 77.44
[0.4701, 0.4340, 0.3832] [0.2845, 0.2733, 0.2805] enabled 77.01

I appreciate any input on this.
Thanks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported evaluation with torch.hub.load('pytorch/vision:v0.10.0', 'inception_v3', pretrained=True), the shown ImageFolder transforms, and both transform_input settings. Compare the measured accuracy with the hub documentation and determine which preprocessing and expected result should be considered correct; done means the discrepancy is explained and the behavior or documentation is aligned.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.