apple / apple/coremltools

ct.convert(convert_to="mlprogram", compute_precision=ct.precision.FLOAT16) have precision problem for some models

Open
#2,603 3 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describing the bug
- I have checked my cases, and found for some models, ct.convert(convert_to="mlprogram", compute_precision=ct.precision.FLOAT16) have big problem of precision, while ct.convert(convert_to="mlprogram", compute_precision=ct.precision.FLOAT32) is correct. also ct.convert(convert_to="neuralnetwork") along with quantization_utils.quantize_weights(model_ct, nbits=16) is correct.

## Stack Trace
-

## To Reproduce
- sorry cannot provide model now

## System environment (please complete the following information):
- coremltools version: 8.3.0
- OS (e.g. MacOS version or Linux type): MacOS_14.5 (23F79)
- Any other relevant version information (e.g. PyTorch or TensorFlow version): pytorch_2.2.2

## Additional context
- https://github.com/apple/coremltools/issues/2301
- https://github.com/apple/coremltools/issues/2481
- and so on

Contributor guide

Open the contributing guide

Research direction

Start at the ct.convert entry point and compare the FLOAT16 and FLOAT32 mlprogram paths with the neuralnetwork plus quantize_weights path described in the report. A reproducible model is needed before the precision difference can be investigated; done means identifying the conversion cause and confirming corrected precision with a regression case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.