apple / apple/coremltools

Large output errors when converting MobileNetV3 in FP16

Open
#2,625 18 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describing the bug
I trained the default torchvision implementation of MobileNetV3Large and convert it to CoreML. In fp32 both models give identical results but when converted to fp16 the CoreML output differs much more than the Torch model. I created a self contained example to show the difference on grayscale images (from 0 to 255), but the errors are equally large on my real data.

## To Reproduce
```python
import asyncio

import coremltools as ct
import matplotlib.pyplot as plt
import numpy as np
import torch
from coremltools.converters.mil.mil import types
from coremltools.models.ml_program.experimental.debugging_utils import MLModelComparator
from PIL import Image
from tabulate import tabulate
from torchvision.models import MobileNet_V3_Large_Weights, mobilenet_v3_large
from tqdm import tqdm

with torch.no_grad():
model_torch32 = mobilenet_v3_large(weights=MobileNet_V3_Large_Weights.DEFAULT)
model_torch32.eval()
model_torch16 = mobilenet_v3_large(weights=MobileNet_V3_Large_Weights.DEFAULT)
model_torch16.eval().to(torch.float16)

model_coreml32 = ct.converters.convert(
model=torch.jit.trace(model_torch32, torch.zeros((1, 3, 224, 224))),
inputs=[ct.ImageType(name="image", shape=(1, 3, 224, 224), scale=1 / 255.0)],
outputs=[ct.TensorType(name="y", dtype=types.fp32)],
minimum_deployment_target=ct.target.iOS18,
compute_units=ct.ComputeUnit.CPU_ONLY,
compute_precision=ct.precision.FLOAT32,
)
model_coreml16 = ct.converters.convert(
model=torch.jit.trace(model_torch32, torch.zeros((1, 3, 224, 224))),
inputs=[ct.ImageType(name="image", shape=(1, 3, 224, 224), scale=1 / 255.0)],
outputs=[ct.TensorType(name="y", dtype=types.fp16)],
minimum_deployment_target=ct.target.iOS18,
compute_units=ct.ComputeUnit.CPU_ONLY,
compute_precision=ct.precision.FLOAT16,
)

# Compare results
xs = list(range(0, 256))
torch32_preds = []
torch16_preds = []
coreml32_preds = []
coreml16_preds = []
for x in tqdm(xs):
x_torch = torch.full((1, 3, 224, 224), x, dtype=torch.float32) / 255.0
torch32_preds.append(model_torch32(x_torch)[0, 0].item())
torch16_preds.append(model_torch16(x_torch.to(torch.float16))[0, 0].item())

x_coreml = Image.fromarray(np.full((224, 224, 3), x, dtype=np.uint8))
coreml32_preds.append(model_coreml32.predict({"image": x_coreml})["y"][0, 0])
coreml16_preds.append(model_coreml16.predict({"image": x_coreml})["y"][0, 0])

# Print error table
print(
tabulate(
[
[
"Torch ",
np.mean(np.abs(np.array(torch32_preds) - np.array(torch16_preds))),
np.mean((np.array(torch32_preds) - np.array(torch16_preds)) ** 2),
],
[
"CoreML",
np.mean(np.abs(np.array(coreml32_preds) - np.array(coreml16_preds))),
np.mean((np.array(coreml32_preds) - np.array(coreml16_preds)) ** 2),
],
],
headers=["Model", "L1", "L2"],
)
)

# plot
plt.plot(xs, torch32_preds, label="Torch32")
plt.plot(xs, torch16_preds, label="Torch16")
plt.plot(xs, coreml32_preds, label="CoreML32", linestyle="dashed")
plt.plot(xs, coreml16_preds, label="CoreML16", linestyle="dashed")
plt.legend()
plt.title("MobileNetV3: Torch vs CoreML")
plt.xlabel("Input pixel value")
plt.ylabel("Output value")
plt.savefig("torch_vs_coreml.png")
plt.close()
```

fp32 vs fp16 comparison for Torch and CoreML:
```
Model L1 Error L2 Error
------- --------- -----------
Torch 0.0164849 0.000468726
CoreML 0.0393638 0.00248957 <-- much larger
```

Comparison plot, CoreML in fp16 output is extremely noisy:
Image

I also tried `MLModelComparator` to find the error but it fails on normal "linear" and "conv" layers (depending on `atol`):
```python
def compare_outputs(operation, reference_output, target_output):
return np.allclose(reference_output, target_output, atol=1e-1)

comparator = MLModelComparator(reference_model=model_coreml32, target_model=model_coreml16, num_predict_intermediate_outputs=720)
failing_ops = asyncio.run(
comparator.find_failing_ops(inputs={"image": x_coreml}, compare_outputs=compare_outputs)
)
print(failing_ops)
```
```
Analyzed operation: classifier_3_weight, type: const: 1%|▏ | 4/720 [00:00<02:18, 5.18it/s]
[type: "linear"
inputs {
key: "x"
value {
arguments {
name: "input_331"
}
}
}
inputs {
key: "weight"
value {
arguments {
name: "classifier_3_weight"
}
}
}
inputs {
key: "bias"
value {
arguments {
name: "classifier_3_bias"
}
}
}
outputs {
name: "y"
type {
tensorType {
dataType: FLOAT32
rank: 2
dimensions {
constant {
size: 1
}
}
dimensions {
constant {
size: 1000
}
}
}
}
}
attributes {
key: "name"
value {
type {
tensorType {
dataType: STRING
}
}
immediateValue {
tensor {
strings {
values: "linear_1"
}
}
}
}
}
]
```

## System environment
coremltools version: 9.0
OS: macOS 26.1
PyTorch version: 2.7.0

Contributor guide

Open the contributing guide

Research direction

Start with the self-contained conversion script, especially the two ct.converters.convert calls and the MLModelComparator comparison. Reproduce the reported FP32/FP16 differences with the stated versions and inspect the failing linear and convolution operations. Done means identifying and correcting the source of the larger CoreML FP16 error, with the comparison showing reduced discrepancy.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.