pytorch convert fail to convert some aten::index operations due to wrong index type
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
## 🐞Describing the bug
The error happens when converting the 2552.
I looked into the details inlined IR and compared it to the debug output.
I found that it caused by this node %2510.
Basically the index does not work.
But didn't know how to reproduce it with simple example yet.
```
%2508 : Long(1, 1, strides=[1, 1], requires_grad=0, device=cpu) = aten::unsqueeze(%2507, %1740), scope: __module.model.proposal_generator # detectron2/detectron2/modeling/proposal_generator/proposal_utils.py:72:0
%2509 : Tensor?[] = prim::ListConstruct(%2508, %topk_idx.1), scope: __module.model.proposal_generator
%2510 : Float(1, 1000, 4, strides=[4000, 4, 1], requires_grad=0, device=cpu) = aten::index(%proposals_i.11, %2509), scope: __module.model.proposal_generator
```
```
INFO:coremltools:Converting op 2550 : unsqueeze
Converting op 2550 : unsqueeze
INFO:coremltools:Adding op '2550' of type expand_dims
Adding op '2550' of type expand_dims
DEBUG:coremltools:Adding const op '2550_axes_0'
Adding const op '2550_axes_0'
INFO:coremltools:Adding op '2550_axes_0' of type const
Adding op '2550_axes_0' of type const
INFO:coremltools:add_op(2550, unsqueeze) done
add_op(2550, unsqueeze) done
INFO:coremltools:Converting op 2551 : listconstruct
Converting op 2551 : listconstruct
INFO:coremltools:add_op(2551, listconstruct) done
add_op(2551, listconstruct) done
INFO:coremltools:Converting op 2552 : index
Converting op 2552 : index
INFO:coremltools:Adding op '2550_broadcasted' of type const
Adding op '2550_broadcasted' of type const
INFO:coremltools:Adding op 'stack_0' of type stack
Adding op 'stack_0' of type stack
DEBUG:coremltools:Adding const op 'stack_0_axis_0'
Adding const op 'stack_0_axis_0'
INFO:coremltools:Adding op 'stack_0_axis_0' of type const
Adding op 'stack_0_axis_0' of type const
...
coremltools/coremltools/converters/mil/frontend/torch/ops.py", line 3492, in index
indices = mb.stack(values=valid_indices, axis=indices_rank)
...
ValueError: Tensors in 'values' of the stack op (stack_0) should share the same data type. Got [.double'>, .int'>]
```
## Stack Trace
```
Traceback (most recent call last):
File "convert2coreml/./export2coreml-detectron2-maskrcnn.py", line 270, in
main()
File "convert2coreml/./export2coreml-detectron2-maskrcnn.py", line 259, in main
export_tracing(
File "convert2coreml/./export2coreml-detectron2-maskrcnn.py", line 222, in export_tracing
mlmodel = ct.converters.convert(
File "convert2coreml/coremltools/coremltools/converters/_converters_entry.py", line 444, in convert
mlmodel = mil_convert(
File "convert2coreml/coremltools/coremltools/converters/mil/converter.py", line 187, in mil_convert
return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
File "convert2coreml/coremltools/coremltools/converters/mil/converter.py", line 211, in _mil_convert
proto, mil_program = mil_convert_to_proto(
File "convert2coreml/coremltools/coremltools/converters/mil/converter.py", line 281, in mil_convert_to_proto
prog = frontend_converter(model, **kwargs)
File "convert2coreml/coremltools/coremltools/converters/mil/converter.py", line 109, in __call__
return load(*args, **kwargs)
File "convert2coreml/coremltools/coremltools/converters/mil/frontend/torch/load.py", line 57, in load
return _perform_torch_convert(converter, debug)
File "convert2coreml/coremltools/coremltools/converters/mil/frontend/torch/load.py", line 96, in _perform_torch_convert
prog = converter.convert()
File "convert2coreml/coremltools/coremltools/converters/mil/frontend/torch/converter.py", line 281, in convert
convert_nodes(self.context, self.graph)
File "convert2coreml/coremltools/coremltools/converters/mil/frontend/torch/ops.py", line 89, in convert_nodes
add_op(context, node)
File "convert2coreml/coremltools/coremltools/converters/mil/frontend/torch/ops.py", line 3492, in index
indices = mb.stack(values=valid_indices, axis=indices_rank)
File "convert2coreml/coremltools/coremltools/converters/mil/mil/ops/registry.py", line 176, in add_op
return cls._add_op(op_cls_to_add, **kwargs)
File "convert2coreml/coremltools/coremltools/converters/mil/mil/builder.py", line 182, in _add_op
new_op.type_value_inference()
File "convert2coreml/coremltools/coremltools/converters/mil/mil/operation.py", line 253, in type_value_inference
output_types = self.type_inference()
File "convert2coreml/coremltools/coremltools/converters/mil/mil/ops/defs/iOS15/tensor_operation.py", line 1262, in type_inference
raise ValueError(msg)
ValueError: Tensors in 'values' of the stack op (stack_0) should share the same data type. Got [.double'>, .int'>].
```
```
%proposals_i.11 : Float(1, 69312, 4, strides=[277248, 4, 1], requires_grad=0, device=cpu) = aten::view(%proposals_i.1, %2183), scope: __module.model.proposal_generator # convert2coreml/detectron2/detectron2/modeling/proposal_generator/rpn.py:527:0
```
Everything looks fine though the proposals_i.11 has float dtype; %2510 try to index it by %2509,
but `%2509 : Tensor?[] = prim::ListConstruct(%2508, %topk_idx.1)` is apparently a list of [long , log] value not a [double, long].
I do not know why coreml think it is [double, long].
And we even [make everything int before hand](https://github.com/apple/coremltools/blob/9065fdc06f929826be3d1c9ded2aad4a89bab32d/coremltools/converters/mil/frontend/torch/ops.py#L3463)
## To Reproduce
does not know how to produce it yet.
## System environment (please complete the following information):
- coremltools version: main head
- ubuntu 22.04
- pytorch 1.13.1
## Additional context
- had this problem when I try to convert detectron2 mask rcnn models.
Contributor guide
Research direction
Start in coremltools/converters/mil/frontend/torch/ops.py around the index implementation at line 3492, and compare its handling of the reported Long index tensors with the stack failure. First reduce the detectron2 Mask R-CNN conversion to a reproducible example; done means the conversion no longer fails when the index values reach the stack operation with mixed types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100