apple / apple/coremltools

adding NMS to an .mlpackage detection model

Open
#1,773 6 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## ❓Question

I have a detection keras model `EfficientDetLite2` That I converted to `mlprogram` as follows:
```
ml_model = ct.convert(keras_model, inputs=[ct.ImageType(name='args_0', shape=(1, 448, 448, 3), scale=1 / 128.0, bias=[-127.0/128.0, -127.0/128.0, -127.0/128.0], channel_first=False)], convert_to = "mlprogram")
ml_model.save("efficientDetLite2.mlpackage")
```
it has two outputs:

- `Identity` representing the boxes_coordinates of shape `(1, 37629, 4)`
- `Identity_1` representing the scores of shape `(1, 37629)`

✅ The model is converted successfully.

Now I wanted to add NMS to the model like follows:
```
coreml_model = ct.models.MLModel("efficientDetLite2.mlpackage")
spec = coreml_model.get_spec()

spec.description.output[0].type.multiArrayType.shape.append(1)
spec.description.output[0].type.multiArrayType.shape.append(37629)
spec.description.output[0].type.multiArrayType.shape.append(4)
spec.description.output[1].type.multiArrayType.shape.append(1)
spec.description.output[1].type.multiArrayType.shape.append(37629)
coreml_model = ct.models.MLModel(spec, weights_dir = coreml_model.weights_dir)

nms_spec = ct.proto.Model_pb2.Model()
nms_spec.specificationVersion = 6

for i in range(2):
model_out = spec.description.output[0 if i == 1 else 1].SerializeToString()
nms_spec.description.input.add()
nms_spec.description.input[i].ParseFromString(model_out)
nms_spec.description.output.add()
nms_spec.description.output[i].ParseFromString(model_out)

nms_spec.description.output[0].name = 'confidence'
nms_spec.description.output[1].name = 'coordinates'

nms = nms_spec.nonMaximumSuppression
nms.confidenceInputFeatureName = "Identity_1"
nms.coordinatesInputFeatureName = "Identity"
nms.confidenceOutputFeatureName = "confidence"
nms.coordinatesOutputFeatureName = "coordinates"
nms.iouThresholdInputFeatureName = "iouThreshold"
nms.confidenceThresholdInputFeatureName = "confidenceThreshold"
nms.pickTop.perClass = True
nms.stringClassLabels.vector.extend(np.array(['object']))
nms.iouThreshold = 0.3
nms.confidenceThreshold = 0.3

output_shapes = [1, 4]
for i in range(2):
ma = nms_spec.description.output[i].type.multiArrayType
ma.shapeRange.sizeRanges.add()
ma.shapeRange.sizeRanges[0].lowerBound = 0
ma.shapeRange.sizeRanges[0].upperBound = -1
ma.shapeRange.sizeRanges.add()
ma.shapeRange.sizeRanges[1].lowerBound = output_shapes[i]
ma.shapeRange.sizeRanges[1].upperBound = output_shapes[i]
del ma.shape[:]

model_nms = ct.models.MLModel(nms_spec)
model_nms.save('nms.mlmodel')
```
✅ NMS model was saved successfully

Now I wanted to combine the models into a pipeline as follows:

```
input_features = [
('args_0', ctdt.Array(448, 448, 3)),
('iouThreshold', ctdt.Double()),
('confidenceThreshold', ctdt.Double())
]
output_features = ['coordinates', 'confidence']
pipeline = ct.models.pipeline.Pipeline(input_features, output_features)
pipeline.spec.specificationVersion = 6

pipeline.add_model(coreml_model)
pipeline.add_model(model_nms)

pipeline.spec.description.input[0].ParseFromString(spec.description.input[0].SerializeToString())
pipeline.spec.description.output[0].ParseFromString(nms_spec.description.output[0].SerializeToString())
pipeline.spec.description.output[1].ParseFromString(nms_spec.description.output[1].SerializeToString())
```
Now i want to save the pipeline as mlprogram

```
model_pipeline = ct.models.MLModel(pipeline.spec, weights_dir = coreml_model.weights_dir)
model_pipeline.save("pipeline.mlpackage")
```
This raises the following warning :
```
/Users/user1/miniconda3/envs/coreml_conv/lib/python3.9/site-packages/coremltools/models/model.py:144: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: compiler error: Encountered an error while compiling a neural network model: at unknown location: Could not open /var/folders/8r/t63s22hs3fn653kxc82_38gc0000gn/T/weights/weight.bin".
_warnings.warn(
```
running the convertion script with `sudo` gives the following warning:
```
/Users/user1/miniconda3/envs/coreml_conv/lib/python3.9/site-packages/coremltools/models/model.py:144: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: compiler error: Encountered an error while compiling a neural network model: at unknown location: Could not open /tmp/weights/weight.bin".
_warnings.warn(
```
In both cases I cant open the model with Xcode 14.1, i get the error `missingMetadataField(named: "inputSchema")`

I tried again with this saving medthod instead:

```
save_spec(pipeline.spec,"pipeline.mlpackage", weights_dir = coreml_model.weights_dir)
```
✅ I get No warnings
✅ I can load the model in Xcode 14.1
❌ I get no detections even with confidenceThreshold = 0.0 .
❌ I can't load the model into Netron.app

I dont know what I did wrong, I am guessing im no passing the weight correctly, but even with wrong weights, I should get detection with 0.0 threhsold.

Contributor guide

Open the contributing guide

Research direction

Start with the conversion script's uses of ct.models.MLModel, Pipeline, and save_spec, focusing on how core_model.weights_dir is passed while combining the models. Compare the generated pipeline in Xcode 14.1 and Netron, then run it with a zero confidence threshold. Done means the saved pipeline loads without warnings and returns detections.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.