apple / apple/coremltools

CoreML Calculation error for some parameter setup case of torch.nn.AvgPool2d

Open
#1,901 8 comments 0 reactions 0 assignees View on GitHub
bug PyTorch (traced) triaged
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describing the bug
Upon converting `torch.nn.AvgPool2d`, CoreML predict (both on host Mac using `libcoremlpython` or on device iPhone 13 Pro) gives erroneous output on parameter setup "kernel size 7, padding 3, stride 4", while the result of some other parameter setup is correct.

## To Reproduce
A minimal code example:
```
#!/usr/bin/env python
# coding: utf-8

import torch.nn
import coremltools as ct

## 1. setup torch module

module_OK_case1 = torch.nn.AvgPool2d(
kernel_size=(3, 3),
stride=(2, 2),
padding=(1, 1),
ceil_mode=False,
count_include_pad=True
)
# case2 is a global pooling for input x (1, 1, 16, 12)
module_OK_case2 = torch.nn.AvgPool2d(
kernel_size=(16, 12),
stride=(1, 1),
padding=(0, 0),
ceil_mode=False,
count_include_pad=True
)
module_bug_reproduce = torch.nn.AvgPool2d(
kernel_size=(7, 7),
stride=(4, 4),
padding=(3, 3),
ceil_mode=False,
count_include_pad=True
)

## 2. choose the model here
# torch_model = module_OK_case1.eval()
# torch_model = module_OK_case2.eval()
torch_model = module_bug_reproduce.eval()

# it appears input shape does not matter
x = torch.randn(1, 1, 16, 12)
x = torch.randn(1, 1, 32, 32)
x = torch.randn(1, 128, 64, 64)

## 3. convert coreml mlmodel

traced_model = torch.jit.trace(torch_model, x)
mlmodel = ct.convert(traced_model, inputs=[ct.TensorType(name="input", shape=x.shape)])

spec = mlmodel.get_spec()
input_name = spec.description.input[0].name
output_name = spec.description.output[0].name

## 4. forward the same tensor with both torch and coreml

y = torch_model(x)
input_dict = { input_name : x }
coreml_out = mlmodel.predict(input_dict)
cy = coreml_out[ output_name ]

## 5. y is from torch, cy is from coreml, the diff sum is expected to be 0

print('Expected 0:', (y - cy).sum())
print('Yet 0:', (y - cy * 7*7).sum()) # 7*7 is for module_bug_reproduce
```

The model conversion succeeds, but there is a numerical mismatch in predictions. My output:
```
Expected 0: tensor(34.4464)
Yet 0: tensor(-0.0048)
```
It seemed that the erroneous output incorrectly divide AvgPool kernel 7x7 twice.

## System environment:
- coremltools version: 6.1
- OS: MacOS Ventura 13.4.1 (22F2083)
- torch: 2.0.1

## Additional context
This problem originates from a network that I was working on, and wrong calculation result is observed on an iPhone 13 Pro.
Later, I confirmed that on host Mac, coremltools `model.predict` yield the same result as iPhone, which is also wrong.

The first major numerical mismatch in the network is from an AvgPool2D layer, whose parameter is kernel 7x7 stride 4x4 padding 3x3, but all the results from previous AvgPool2D with other parameter is correct.

It seemed a bug for CoreML framework instead of `coremltools`. But I think:
1. It's still worth to report it here for others may come across.
2. It's possible the bug is related with `coremltools` because I'm not sure what's converted is correct or not.
3. (Update) Already report to Apple CoreML, whose bug report website lacks a rich text editor, so that bug report has a link here for good readability.

Contributor guide

Open the contributing guide

Research direction

Start with the supplied minimal Python reproduction using torch.nn.AvgPool2d, torch.jit.trace, and ct.convert, then compare the PyTorch and Core ML predictions for the stated parameters. Determine whether the mismatch is introduced during conversion or by the Core ML runtime; done means identifying the responsible side and confirming matching output or a documented limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.