Unable to map torch_upsample_nearest_neighbor to core upsample, using flexible input shapes during conversion
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
## 🐞Describing the bug
I get the error `Unable to map torch_upsample_nearest_neighbor to core upsample` when I try to convert the DETR PyTorch model. I tried to go deep into the package to see that the issue is arising from the `_try_get_upsample_factor` function where the `op.op_type` is `gather` but the conditional checks for `cast`.
## Stack Trace
Traceback (most recent call last):
File "test.py", line 19, in
model = ct.convert(
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/_converters_entry.py", line 444, in convert
mlmodel = mil_convert(
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/converter.py", line 190, in mil_convert
return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/converter.py", line 217, in _mil_convert
proto, mil_program = mil_convert_to_proto(
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/converter.py", line 282, in mil_convert_to_proto
prog = frontend_converter(model, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/converter.py", line 112, in __call__
return load(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/frontend/torch/load.py", line 57, in load
return _perform_torch_convert(converter, debug)
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/frontend/torch/load.py", line 96, in _perform_torch_convert
prog = converter.convert()
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/frontend/torch/converter.py", line 300, in convert
self.torch_passes(prog)
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/frontend/torch/ssa_passes/torch_passes.py", line 24, in torch_passes
PASS_REGISTRY[p](prog)
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/mil/passes/graph_pass.py", line 14, in __call__
self.apply(prog)
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/frontend/torch/ssa_passes/torch_upsample_to_core_upsample.py", line 35, in apply
_torch_upsample_to_core_upsample_block(f)
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/mil/passes/helper.py", line 42, in wrapper
return func(*args)
File "/usr/local/lib/python3.8/dist-packages/coremltools/converters/mil/frontend/torch/ssa_passes/torch_upsample_to_core_upsample.py", line 47, in _torch_upsample_to_core_upsample_block
raise ValueError("Unable to map {} to core upsample".format(op.op_type))
ValueError: Unable to map torch_upsample_nearest_neighbor to core upsample
# Python code snippet
```
from transformers import DetrFeatureExtractor, DetrForObjectDetection
import torch
from PIL import Image
import requests
import coremltools as ct
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
image_processor = DetrFeatureExtractor.from_pretrained("facebook/detr-resnet-50")
model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50", return_dict=False)
inputs = image_processor(images=image, return_tensors="pt")
outputs = model(**inputs)
traced_model = torch.jit.trace(model, example_inputs=inputs["pixel_values"])
model = ct.convert(
traced_model,
convert_to="mlprogram",
inputs=[ct.ImageType(shape=(1, 3, ct.RangeDim(256, 3072), ct.RangeDim(256, 3072)))]
)
```
## System environment (please complete the following information):
- coremltools version: 6.1
- OS: Ubuntu 20.04.4 LTS
- PyTorch version: 1.10.0
- transformers version: 4.19.3
Contributor guide
Research direction
Start in coremltools/converters/mil/frontend/torch/ssa_passes/torch_upsample_to_core_upsample.py, especially _try_get_upsample_factor and the conditional handling op.op_type. Reproduce the failure with the DETR conversion snippet and flexible RangeDim input shapes. Done means the traced DETR model converts without the unable-to-map error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100