apple / apple/coremltools

pytorch convert fail to convert some aten::index operations due to wrong index type

Open
#1,786 3 comments 0 reactions 0 assignees View on GitHub
bug PyTorch (traced)
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describing the bug

The error happens when converting the 2552.
I looked into the details inlined IR and compared it to the debug output.
I found that it caused by this node %2510.
Basically the index does not work.

But didn't know how to reproduce it with simple example yet.

```
%2508 : Long(1, 1, strides=[1, 1], requires_grad=0, device=cpu) = aten::unsqueeze(%2507, %1740), scope: __module.model.proposal_generator # detectron2/detectron2/modeling/proposal_generator/proposal_utils.py:72:0
%2509 : Tensor?[] = prim::ListConstruct(%2508, %topk_idx.1), scope: __module.model.proposal_generator
%2510 : Float(1, 1000, 4, strides=[4000, 4, 1], requires_grad=0, device=cpu) = aten::index(%proposals_i.11, %2509), scope: __module.model.proposal_generator
```

```
INFO:coremltools:Converting op 2550 : unsqueeze
Converting op 2550 : unsqueeze
INFO:coremltools:Adding op '2550' of type expand_dims
Adding op '2550' of type expand_dims
DEBUG:coremltools:Adding const op '2550_axes_0'
Adding const op '2550_axes_0'
INFO:coremltools:Adding op '2550_axes_0' of type const
Adding op '2550_axes_0' of type const
INFO:coremltools:add_op(2550, unsqueeze) done
add_op(2550, unsqueeze) done
INFO:coremltools:Converting op 2551 : listconstruct
Converting op 2551 : listconstruct
INFO:coremltools:add_op(2551, listconstruct) done
add_op(2551, listconstruct) done
INFO:coremltools:Converting op 2552 : index
Converting op 2552 : index
INFO:coremltools:Adding op '2550_broadcasted' of type const
Adding op '2550_broadcasted' of type const
INFO:coremltools:Adding op 'stack_0' of type stack
Adding op 'stack_0' of type stack
DEBUG:coremltools:Adding const op 'stack_0_axis_0'
Adding const op 'stack_0_axis_0'
INFO:coremltools:Adding op 'stack_0_axis_0' of type const
Adding op 'stack_0_axis_0' of type const
...
coremltools/coremltools/converters/mil/frontend/torch/ops.py", line 3492, in index
indices = mb.stack(values=valid_indices, axis=indices_rank)
...
ValueError: Tensors in 'values' of the stack op (stack_0) should share the same data type. Got [.double'>, .int'>]
```
## Stack Trace

```
Traceback (most recent call last):
File "convert2coreml/./export2coreml-detectron2-maskrcnn.py", line 270, in
main()
File "convert2coreml/./export2coreml-detectron2-maskrcnn.py", line 259, in main
export_tracing(
File "convert2coreml/./export2coreml-detectron2-maskrcnn.py", line 222, in export_tracing
mlmodel = ct.converters.convert(
File "convert2coreml/coremltools/coremltools/converters/_converters_entry.py", line 444, in convert
mlmodel = mil_convert(
File "convert2coreml/coremltools/coremltools/converters/mil/converter.py", line 187, in mil_convert
return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
File "convert2coreml/coremltools/coremltools/converters/mil/converter.py", line 211, in _mil_convert
proto, mil_program = mil_convert_to_proto(
File "convert2coreml/coremltools/coremltools/converters/mil/converter.py", line 281, in mil_convert_to_proto
prog = frontend_converter(model, **kwargs)
File "convert2coreml/coremltools/coremltools/converters/mil/converter.py", line 109, in __call__
return load(*args, **kwargs)
File "convert2coreml/coremltools/coremltools/converters/mil/frontend/torch/load.py", line 57, in load
return _perform_torch_convert(converter, debug)
File "convert2coreml/coremltools/coremltools/converters/mil/frontend/torch/load.py", line 96, in _perform_torch_convert
prog = converter.convert()
File "convert2coreml/coremltools/coremltools/converters/mil/frontend/torch/converter.py", line 281, in convert
convert_nodes(self.context, self.graph)
File "convert2coreml/coremltools/coremltools/converters/mil/frontend/torch/ops.py", line 89, in convert_nodes
add_op(context, node)
File "convert2coreml/coremltools/coremltools/converters/mil/frontend/torch/ops.py", line 3492, in index
indices = mb.stack(values=valid_indices, axis=indices_rank)
File "convert2coreml/coremltools/coremltools/converters/mil/mil/ops/registry.py", line 176, in add_op
return cls._add_op(op_cls_to_add, **kwargs)
File "convert2coreml/coremltools/coremltools/converters/mil/mil/builder.py", line 182, in _add_op
new_op.type_value_inference()
File "convert2coreml/coremltools/coremltools/converters/mil/mil/operation.py", line 253, in type_value_inference
output_types = self.type_inference()
File "convert2coreml/coremltools/coremltools/converters/mil/mil/ops/defs/iOS15/tensor_operation.py", line 1262, in type_inference
raise ValueError(msg)
ValueError: Tensors in 'values' of the stack op (stack_0) should share the same data type. Got [.double'>, .int'>].
```

```
%proposals_i.11 : Float(1, 69312, 4, strides=[277248, 4, 1], requires_grad=0, device=cpu) = aten::view(%proposals_i.1, %2183), scope: __module.model.proposal_generator # convert2coreml/detectron2/detectron2/modeling/proposal_generator/rpn.py:527:0
```

Everything looks fine though the proposals_i.11 has float dtype; %2510 try to index it by %2509,
but `%2509 : Tensor?[] = prim::ListConstruct(%2508, %topk_idx.1)` is apparently a list of [long , log] value not a [double, long].

I do not know why coreml think it is [double, long].
And we even [make everything int before hand](https://github.com/apple/coremltools/blob/9065fdc06f929826be3d1c9ded2aad4a89bab32d/coremltools/converters/mil/frontend/torch/ops.py#L3463)

## To Reproduce

does not know how to produce it yet.

## System environment (please complete the following information):
- coremltools version: main head
- ubuntu 22.04
- pytorch 1.13.1

## Additional context
- had this problem when I try to convert detectron2 mask rcnn models.

Contributor guide

Open the contributing guide

Research direction

Start in coremltools/converters/mil/frontend/torch/ops.py around the index implementation at line 3492, and compare its handling of the reported Long index tensors with the stack failure. First reduce the detectron2 Mask R-CNN conversion to a reproducible example; done means the conversion no longer fails when the index values reach the stack operation with mixed types.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.