mobilebert fails returns bad results on CoreML
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
🐛 Describe the bug
Result is nan on FP16
sh .ci/scripts/test_model.sh mobilebert cmake coreml-pybind-float16
And result has high MSE with eager on FP32:
sh .ci/scripts/test_model.sh mobilebert cmake coreml-pybind-float32
Versions
Collecting environment information...
PyTorch version: 2.9.0.dev20250725
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 15.5 (arm64)
GCC version: Could not collect
Clang version: 17.0.0 (clang-1700.0.13.5)
CMake version: version 3.31.6
Libc version: N/A
Python version: 3.10.18 (main, Jun 5 2025, 08:37:47) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-15.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Apple M1 Pro
Versions of relevant libraries:
[pip3] executorch==0.8.0a0+8e6c8ff
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.2.6
[pip3] optimum-executorch==0.0.0.dev0
[pip3] pytorch_tokenizers==0.1.0
[pip3] torch==2.9.0.dev20250725
[pip3] torchao==0.13.0+git2eb4f9762
[pip3] torchaudio==2.8.0.dev20250725
[pip3] torchdata==0.11.0
[pip3] torchsr==1.0.4
[pip3] torchtune==0.6.1
[pip3] torchvision==0.24.0.dev20250725
[conda] executorch 0.8.0a0+8e6c8ff pypi_0 pypi
[conda] numpy 2.2.6 pypi_0 pypi
[conda] optimum-executorch 0.0.0.dev0 pypi_0 pypi
[conda] pytorch-tokenizers 0.1.0 pypi_0 pypi
[conda] torch 2.9.0.dev20250725 pypi_0 pypi
[conda] torchao 0.13.0+git2eb4f9762 pypi_0 pypi
[conda] torchaudio 2.8.0.dev20250725 pypi_0 pypi
[conda] torchdata 0.11.0 pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchtune 0.6.1 pypi_0 pypi
[conda] torchvision 0.24.0.dev20250725 pypi_0 pypi
cc @kimishpatel @YifanShenSZ @cymbalrush
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the two .ci/scripts/test_model.sh commands for mobilebert and compare the CoreML FP16 and FP32 results with eager execution. Trace the mobilebert CoreML conversion or runtime path implicated by those commands. Done means FP16 no longer returns NaN and FP32 no longer has high MSE against eager.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100