adding NMS to an .mlpackage detection model
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
## ❓Question
I have a detection keras model `EfficientDetLite2` That I converted to `mlprogram` as follows:
```
ml_model = ct.convert(keras_model, inputs=[ct.ImageType(name='args_0', shape=(1, 448, 448, 3), scale=1 / 128.0, bias=[-127.0/128.0, -127.0/128.0, -127.0/128.0], channel_first=False)], convert_to = "mlprogram")
ml_model.save("efficientDetLite2.mlpackage")
```
it has two outputs:
- `Identity` representing the boxes_coordinates of shape `(1, 37629, 4)`
- `Identity_1` representing the scores of shape `(1, 37629)`
✅ The model is converted successfully.
Now I wanted to add NMS to the model like follows:
```
coreml_model = ct.models.MLModel("efficientDetLite2.mlpackage")
spec = coreml_model.get_spec()
spec.description.output[0].type.multiArrayType.shape.append(1)
spec.description.output[0].type.multiArrayType.shape.append(37629)
spec.description.output[0].type.multiArrayType.shape.append(4)
spec.description.output[1].type.multiArrayType.shape.append(1)
spec.description.output[1].type.multiArrayType.shape.append(37629)
coreml_model = ct.models.MLModel(spec, weights_dir = coreml_model.weights_dir)
nms_spec = ct.proto.Model_pb2.Model()
nms_spec.specificationVersion = 6
for i in range(2):
model_out = spec.description.output[0 if i == 1 else 1].SerializeToString()
nms_spec.description.input.add()
nms_spec.description.input[i].ParseFromString(model_out)
nms_spec.description.output.add()
nms_spec.description.output[i].ParseFromString(model_out)
nms_spec.description.output[0].name = 'confidence'
nms_spec.description.output[1].name = 'coordinates'
nms = nms_spec.nonMaximumSuppression
nms.confidenceInputFeatureName = "Identity_1"
nms.coordinatesInputFeatureName = "Identity"
nms.confidenceOutputFeatureName = "confidence"
nms.coordinatesOutputFeatureName = "coordinates"
nms.iouThresholdInputFeatureName = "iouThreshold"
nms.confidenceThresholdInputFeatureName = "confidenceThreshold"
nms.pickTop.perClass = True
nms.stringClassLabels.vector.extend(np.array(['object']))
nms.iouThreshold = 0.3
nms.confidenceThreshold = 0.3
output_shapes = [1, 4]
for i in range(2):
ma = nms_spec.description.output[i].type.multiArrayType
ma.shapeRange.sizeRanges.add()
ma.shapeRange.sizeRanges[0].lowerBound = 0
ma.shapeRange.sizeRanges[0].upperBound = -1
ma.shapeRange.sizeRanges.add()
ma.shapeRange.sizeRanges[1].lowerBound = output_shapes[i]
ma.shapeRange.sizeRanges[1].upperBound = output_shapes[i]
del ma.shape[:]
model_nms = ct.models.MLModel(nms_spec)
model_nms.save('nms.mlmodel')
```
✅ NMS model was saved successfully
Now I wanted to combine the models into a pipeline as follows:
```
input_features = [
('args_0', ctdt.Array(448, 448, 3)),
('iouThreshold', ctdt.Double()),
('confidenceThreshold', ctdt.Double())
]
output_features = ['coordinates', 'confidence']
pipeline = ct.models.pipeline.Pipeline(input_features, output_features)
pipeline.spec.specificationVersion = 6
pipeline.add_model(coreml_model)
pipeline.add_model(model_nms)
pipeline.spec.description.input[0].ParseFromString(spec.description.input[0].SerializeToString())
pipeline.spec.description.output[0].ParseFromString(nms_spec.description.output[0].SerializeToString())
pipeline.spec.description.output[1].ParseFromString(nms_spec.description.output[1].SerializeToString())
```
Now i want to save the pipeline as mlprogram
```
model_pipeline = ct.models.MLModel(pipeline.spec, weights_dir = coreml_model.weights_dir)
model_pipeline.save("pipeline.mlpackage")
```
This raises the following warning :
```
/Users/user1/miniconda3/envs/coreml_conv/lib/python3.9/site-packages/coremltools/models/model.py:144: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: compiler error: Encountered an error while compiling a neural network model: at unknown location: Could not open /var/folders/8r/t63s22hs3fn653kxc82_38gc0000gn/T/weights/weight.bin".
_warnings.warn(
```
running the convertion script with `sudo` gives the following warning:
```
/Users/user1/miniconda3/envs/coreml_conv/lib/python3.9/site-packages/coremltools/models/model.py:144: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: compiler error: Encountered an error while compiling a neural network model: at unknown location: Could not open /tmp/weights/weight.bin".
_warnings.warn(
```
In both cases I cant open the model with Xcode 14.1, i get the error `missingMetadataField(named: "inputSchema")`
I tried again with this saving medthod instead:
```
save_spec(pipeline.spec,"pipeline.mlpackage", weights_dir = coreml_model.weights_dir)
```
✅ I get No warnings
✅ I can load the model in Xcode 14.1
❌ I get no detections even with confidenceThreshold = 0.0 .
❌ I can't load the model into Netron.app
I dont know what I did wrong, I am guessing im no passing the weight correctly, but even with wrong weights, I should get detection with 0.0 threhsold.
Contributor guide
Research direction
Start with the conversion script's uses of ct.models.MLModel, Pipeline, and save_spec, focusing on how core_model.weights_dir is passed while combining the models. Compare the generated pipeline in Xcode 14.1 and Netron, then run it with a zero confidence threshold. Done means the saved pipeline loads without warnings and returns detections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100