apple / apple/coremltools

Converted classification model returns invalid confidence scores (e.g. negative numbers and Nan)

Open
#1,015 1 comment 0 reactions 0 assignees View on GitHub
awaiting response bug tf1.x
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describe the bug
I converted a TensorFlow graph into a mlmodel. I can correctly deploy the mlmodel file except for the fact that the values it returns do not make any sense. It's a classification model supposed to return confidence scores in [0,1], but it returns arbitrary large and/or small numbers. Once, I even got a NaN value.

The original TensorFlow graph works just fine. It feels like maybe the preprocessing isn't applied correctly? I have configured the preprocessing during the conversion though.

## To Reproduce
This is how I converted the model:
```python

bias = -mean / std
image_input = ct.ImageType(shape=(1, input_size, input_size, 3,),
bias=[bias, bias, bias],
scale=1 / std,
name=input_feature_name,
)

ml_model = ct.convert(
model=model_path,
inputs=[image_input],
outputs=[output_feature_name],
minimum_deployment_target=ct.target.iOS13,
)

ml_model.save(output_path)
```

I'm sorry, I don't have model file which I'm allowed to share. However, I could reproduce this with multiple models (e.g. Inceptionv3 and MobileNetv2).

## System environment (please complete the following information):
- coremltools version 4.0:
- OS (e.g., MacOS, Linux): Ubuntu 18
- How you install python (anaconda, virtualenv, system): virtualenv
- python version (e.g. 3.7): 3.7
- any other relevant information:
- TensorFlow: 1.15

## Additional context
My mlmodel integration within my iOS app isn't a new one. I originally set it up to target iOS 11 along with the following successful setup:
- tfcoreml 1.0
- TensorFlow 1.14
- coremltools 3.1 (not sure about this, wasn't added explicitly, but used by tfcoreml)

That's working as it should. It works with older versions of the same graphs, which are failing now, too. However, before retraining my model I migrated from TensorFlow 1.14 to 1.15 and so [I couldn't use that setup anymore](https://github.com/tf-coreml/tf-coreml/issues/336).

As an intermediate solution I also tried to use `tfcoreml 1.1` with `coremltools 3.4` to target iOS 13. It suffers the same issue as the finally used setup of `coremltools 4.0`.

I got no clue how to continue debugging this. Any help is appreciated very much!

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the conversion with the shown ImageType preprocessing and iOS 13 target, using Inceptionv3 or MobileNetv2 if available. Compare the converted model's confidence outputs with the original TensorFlow graph across the listed TensorFlow, coremltools, and tfcoreml versions; done means scores remain in [0,1] without NaN values.

Written by the indexing model from the issue text.

Assessment

Tech stack
ios, python, tensorflow
Domain
machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.