apple / apple/coremltools

torch d2go model to coreml conversion - Unknown error in building network shapes

Open
#1,505 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describe the bug

Attempting to convert the pytorch based d2go maskrcnn model with FBNETV2C4Backbone (https://github.com/facebookresearch/d2go/blob/main/configs/mask_rcnn_fbnetv3a_C4.yaml) to coreml.

Conversion to MIL successful
```
Converting Frontend ==> MIL Ops: 100%|█████████▉| 1027/1029 [00:04<00:00, 217.11 ops/s]
```
with a number of torch ops needing to be added, and some with minor patches (I'll list these at the end).

However the model runs into an unknown error when compiling the model `"compiler error: Unknown error in building network shapes."`

I've noticed if I convert the model with the `convert_to=mlprogram` that the error changes to `"compiler error: Encountered an error while compiling a neural network model: in operation of type cond: cond must return same types from both branches"`

## Stack Trace
```
Translating MIL ==> NeuralNetwork Ops: 100%|██████████| 23/23 [00:00<00:00, 12615.27 ops/s]
Translating MIL ==> NeuralNetwork Ops: 100%|██████████| 1030/1030 [00:13<00:00, 76.75 ops/s]
/conversion/lib/python3.9/site-packages/coremltools/models/model.py:137: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: Unknown error in building network shapes.".
```

but changes to
```
RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: Encountered an error while compiling a neural network model: in operation of type cond: cond must return same types from both branches".
```

when using `convert_to=mlprogram` when converting the model.

## To Reproduce

I've not been able to isolate the layer causing this issue in the translation of MIL to NN, but would be very happy to dig deeper and try and find a minimal example. IF there are some suggestions on how I might be able to do this that would be rerally appreciated.

I'm running a basic conversion script

```
import coremltools as ct
import os
import torch
import torchvision
from d2go.model_zoo import model_zoo
from detectron2.export import TracingAdapter

import maskrcnn.ops

def inference_func(tmodel, image):
inputs= [{"image": image}]
return tmodel.inference(inputs, do_postprocess=False)[0]

def main():
model = model_zoo.get('mask_rcnn_fbnetv3a_C4.yaml', trained=True)

example = torch.rand(3, 224, 224)
wrapper = TracingAdapter(model, example, inference_func)
wrapper.eval()

traced_model = torch.jit.trace(wrapper, (example,))

inputs = [ct.ImageType(name="my_input", shape=(3, 224, 224))]
coreml_model = ct.convert(
traced_model,
inputs=inputs
)

if __name__ == '__main__':
main()

```

## System environment (please complete the following information):

- python dependencies
```
- python=3.9
- pytorch=1.9.1
- torchvision=0.10.1
- coremltools=5.2.0
```
- MacOS 12.3.1

## Additional context

added torch ops
```
nms
repeat_interleave
numel
roi_align
logicaland
```

ops with minor patches
```
clamp
narrow
index- handle broadcasting indexs
max - inputs are len == 1
split - handle case when num_splits == 1
to - handle the case where the dtype is not set, this should be inferred from the Tensor dtype. see, https://pytorch.org/docs/stable/generated/torch.Tensor.to.html?highlight=#torch.Tensor.to
tupleunpack - handle case when len(output) == 1
```

Contributor guide

Open the contributing guide

Research direction

Start with the provided conversion script, the traced d2go Mask R-CNN model, and the ct.convert call using the MIL-to-NeuralNetwork and mlprogram paths. Reproduce the compiler errors, then isolate the failing operation or minimal model; done means identifying a reproducible conversion failure and a focused location for the fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.