Cannot obtain the accuracy stated in the doc for inception_v3 pretrained on Imagenet
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 7.3k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 13
Description
Hi.
I'm trying to evaluate inception_v3 pretrained model from the hub on Imagenet (ILSVRC 2012) test set. I use the following evaluation code:
def compute_accuracy(self):
num_correct = 0
num_images = 0
_IMAGE_MEAN_VALUE = [0.485, 0.456, 0.406]
_IMAGE_STD_VALUE = [0.229, 0.224, 0.225]
imgnet_loader = torch.utils.data.DataLoader(
ImageFolder('/home/amin/dataset/ILSVRC/val',
transforms.Compose([
transforms.Resize(299),
transforms.CenterCrop(299),
transforms.ToTensor(),
transforms.Normalize(mean=_IMAGE_MEAN_VALUE, std=_IMAGE_STD_VALUE),
])),
batch_size=16, shuffle=True,
num_workers=8, pin_memory=True)
self.model = torch.hub.load('pytorch/vision:v0.10.0', 'inception_v3', pretrained=True).cuda()
self.model.eval()
for i, (images, targets) in \
enumerate(tqdm(imgnet_loader, desc="Compute Accuracy", total=len(imgnet_loader))):
images = images.cuda() if torch.cuda.is_available() else images.cpu()
targets = targets.cuda() if torch.cuda.is_available() else targets.cpu()
output_dict = self.model(images)
pred=output_dict.argmax(dim=1)
num_correct += (pred == targets).sum().item()
num_images += images.size(0)
classification_acc = num_correct / float(num_images) * 100
return classification_acc
However, the accuracy I get is 77.216 while it should be 77.45 according to this page. I figured that the model has a transform_input as preprocess in itself. So if we are doing normalization beforehand (as suggested in the example code), we should set transform_input to false. So if I add self.model.transform_input = False, I get 77.472, Which is closer to the expected value but not exactly the same.
Assuming that the issue isn't from my code, I also found the thread on std, mean values (#1439). So I tested some of the values suggested there as well, and got these results:
| mean | std | model's transform_input | Accuracy |
|---|---|---|---|
| [0.485, 0.456, 0.406] | [0.229, 0.224, 0.225] | disabled | 77.472 |
| [0.485, 0.456, 0.406] | [0.229, 0.224, 0.225] | enabled | 77.216 |
| [0.4803, 0.4569, 0.4083] | [0.2806, 0.2736, 0.2877] | disabled | 77.448 |
| [0.4803, 0.4569, 0.4083] | [0.2806, 0.2736, 0.2877] | enabled | 76.986 |
| [0.4845, 0.4541, 0.4025] | [0.2724, 0.2637, 0.2761] | disabled | 77.456 |
| [0.4845, 0.4541, 0.4025] | [0.2724, 0.2637, 0.2761] | enabled | 77.03 |
| [0.4701, 0.4340, 0.3832] | [0.2845, 0.2733, 0.2805] | disabled | 77.44 |
| [0.4701, 0.4340, 0.3832] | [0.2845, 0.2733, 0.2805] | enabled | 77.01 |
I appreciate any input on this.
Thanks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported evaluation with torch.hub.load('pytorch/vision:v0.10.0', 'inception_v3', pretrained=True), the shown ImageFolder transforms, and both transform_input settings. Compare the measured accuracy with the hub documentation and determine which preprocessing and expected result should be considered correct; done means the discrepancy is explained and the behavior or documentation is aligned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100