pytorch / pytorch/executorch

mobilebert fails returns bad results on CoreML

Open
#12,907 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

module: coreml
Dominant language
Python
Stars
5k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
581

Description

🐛 Describe the bug

Result is nan on FP16

sh .ci/scripts/test_model.sh mobilebert cmake coreml-pybind-float16

And result has high MSE with eager on FP32:

sh .ci/scripts/test_model.sh mobilebert cmake coreml-pybind-float32
Versions

Collecting environment information...
PyTorch version: 2.9.0.dev20250725
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 15.5 (arm64)
GCC version: Could not collect
Clang version: 17.0.0 (clang-1700.0.13.5)
CMake version: version 3.31.6
Libc version: N/A

Python version: 3.10.18 (main, Jun 5 2025, 08:37:47) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-15.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] executorch==0.8.0a0+8e6c8ff
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.2.6
[pip3] optimum-executorch==0.0.0.dev0
[pip3] pytorch_tokenizers==0.1.0
[pip3] torch==2.9.0.dev20250725
[pip3] torchao==0.13.0+git2eb4f9762
[pip3] torchaudio==2.8.0.dev20250725
[pip3] torchdata==0.11.0
[pip3] torchsr==1.0.4
[pip3] torchtune==0.6.1
[pip3] torchvision==0.24.0.dev20250725
[conda] executorch 0.8.0a0+8e6c8ff pypi_0 pypi
[conda] numpy 2.2.6 pypi_0 pypi
[conda] optimum-executorch 0.0.0.dev0 pypi_0 pypi
[conda] pytorch-tokenizers 0.1.0 pypi_0 pypi
[conda] torch 2.9.0.dev20250725 pypi_0 pypi
[conda] torchao 0.13.0+git2eb4f9762 pypi_0 pypi
[conda] torchaudio 2.8.0.dev20250725 pypi_0 pypi
[conda] torchdata 0.11.0 pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchtune 0.6.1 pypi_0 pypi
[conda] torchvision 0.24.0.dev20250725 pypi_0 pypi

cc @kimishpatel @YifanShenSZ @cymbalrush

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the two .ci/scripts/test_model.sh commands for mobilebert and compare the CoreML FP16 and FP32 results with eager execution. Trace the mobilebert CoreML conversion or runtime path implicated by those commands. Done means FP16 no longer returns NaN and FP32 no longer has high MSE against eager.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.