Cannot convert torchvision models that have multi-dimensional dictionary outputs (even with a wrapper).
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
## 🐞Describing the bug
- Make sure to only create an issue here for bugs in the coremltools Python package. If this is a bug with the Core ML Framework or Xcode, please submit your bug here: https://developer.apple.com/bug-reporting/
- Provide a clear and consise description of the bug.
Cannot convert torchvision models that have a complicated output, even if a wrapper is used to make the output into a tensor)
## Stack Trace
- If applicable, please paste the complete stack trace.
[/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/mil/types/type_tensor.py](https://localhost:8080/#) in val(self, v)
86 def val(self, v):
87 if not isinstance(v, np.ndarray):
---> 88 raise ValueError(
89 "tensor should have value of type ndarray, got {} instead".format(
90 type(v)
ValueError: tensor should have value of type ndarray, got instead
## To Reproduce
- Please add a minimal code example that can reproduce the error when running it.
```
import torchvision
import torch
import torch.nn as nn
import coremltools as ct
model = torchvision.models.detection.maskrcnn_resnet50_fpn()
image = torch.randn(1,3,400,400)
class Wrapped(nn.Module):
def __init__(self):
super(Wrapped, self).__init__()
self.model = model.eval()
def forward(self, x):
res = self.model(x)
# Extract the tensor we want from the output dictionary
x = res[0]['boxes']
return x
new_model = Wrapped().eval()
traced_model = torch.jit.trace(new_model,image)
mlmodel = ct.convert(traced_model,
inputs=[ct.TensorType(shape=[1,3, 400, 400])],
source="pytorch",
convert_to="mlprogram")
```
- If the model conversion succeeds, but there is a numerical mismatch in predictions, please include the code used for comparisons.
## System environment (please complete the following information):
- coremltools version: 6.1
- OS (e.g. MacOS version or Linux type): MacOS and Linux
- Any other relevant version information (e.g. PyTorch or TensorFlow version): pytorch==1.12.1 torchvision==0.13.1
## Additional context
- Add anything else about the problem here that you want to share.
Contributor guide
Research direction
Start by reproducing the torchvision Mask R-CNN wrapper conversion with the provided PyTorch and coremltools versions. Trace the failure through the type_tensor.py val method and the ct.convert entry point. Done means the wrapped model converts successfully to an mlprogram without the numpy.float32 tensor validation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100