apple / apple/coremltools

Converting RTDETR Object Detection Model to CoreML

Open
#2,440 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

I am trying to convert the RTDETR Object Detection model to CoreML for use in an iOS app. However, I am encountering a data type mismatch error during the conversion process.

The code i am using:

```
import torch
import coremltools as ct
from transformers import RTDetrForObjectDetection

model = RTDetrForObjectDetection.from_pretrained("TMP-finetune/runs/Jan22_16-22-29")

example_input = (torch.rand(1, 3, 512, 512),)

# Generate ExportedProgram by exporting
exported_model = torch.export.export(model, example_input)

input_images = ct.ImageType(name="images", shape=example_input[0].shape)

# Convert the model
mlmodel = ct.convert(
exported_model,
inputs=[input_images]
)

# Save the Core ML model
mlmodel.save("best.mlmodel")
```

The error i am getting:

![Image](https://github.com/user-attachments/assets/e2dc48f3-cd6e-490b-be79-f95527996f88)

I have tried searching for solutions but haven't found a way to resolve this data type mismatch issue. Any guidance or suggestions would be greatly appreciated.

I have also opened a related issue on another repository, which can be found here: https://github.com/huggingface/transformers/issues/35905#issuecomment-2636625966

Thanks in advance

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure in the shown torch.export.export and ct.convert calls with RTDetrForObjectDetection, then inspect the reported data type mismatch. Compare the conversion behavior with the linked Transformers issue. Done means identifying a compatible conversion path and successfully saving best.mlmodel for iOS use.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.