apple / apple/coremltools

Error converting from torch jit trace

Open
#1,861 1 comment 0 reactions 0 assignees View on GitHub
bug PyTorch (traced) triaged
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

```python
import torch
import numpy as np
from transformers import AutoModel, AutoProcessor, OwlViTModel, OwlViTProcessor, CLIPTokenizerFast, CLIPTokenizer
import coremltools as ct
from PIL import Image
import requests

class MyOpenDetector(torch.nn.Module):
def __init__(self, model=None):
super(MyOpenDetector, self).__init__()
self.model = model

def forward(self, input_ids, pixel_values, attention_mask):
# inputs = {"input_ids":x[0], "attention_mask":x[1], "pixel_values":x[2]}
outputs = self.model(input_ids=input_ids, pixel_values=pixel_values, attention_mask=attention_mask)
logits_per_image = outputs[0] # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities

return probs

def save_owlvitmodel(inputs, modelname):

openModel = AutoModel.from_pretrained(modelname, torchscript=True).eval()

x = tuple([inputs['input_ids'], inputs['pixel_values'], inputs['attention_mask']])
model = MyOpenDetector(model=openModel)
traced_model = torch.jit.trace(model, x)
torch.jit.save(traced_model, 'traced_owlvit.pt')

return traced_model

modelname = "google/owlvit-base-patch32"
processor = AutoProcessor.from_pretrained(modelname, torchscript=True)
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(text=[["a photo of a cat", "a photo of a dog"]], images=torch.Tensor(np.asarray(image)), return_tensors="pt")

traced_model = save_owlvitmodel(inputs, modelname)

loaded_model = torch.jit.load("traced_owlvit.pt")
loaded_model.eval()

x = tuple([inputs['input_ids'], inputs['pixel_values'], inputs['attention_mask']])
probs = loaded_model(*x)

mlmodel = ct.convert(
traced_model,
inputs=[ct.TensorType(name="input_ids", shape=(ct.RangeDim(1, 16),16), dtype=np.int32),
ct.TensorType(name="pixel_values", shape=(ct.RangeDim(1, 3),3,768,768), dtype=np.float32),
ct.TensorType(name="attention_mask", shape=(ct.RangeDim(1, 16),16), dtype=np.int32),],
)
mlmodel.save('coremlmodel_owlvit.mlmodel')
```
## System environment (please complete the following information):
- coremltools version:
- Ubuntu 20
- torch==2.0.0

Contributor guide

Open the contributing guide

Research direction

Run the supplied reproduction script on Ubuntu 20 with the stated PyTorch version, first filling in the missing coremltools version and capturing the complete error from ct.convert. Inspect the torch.jit.trace output and the three TensorType input declarations to isolate the failing conversion path. Done means the failure is reproducible and conversion succeeds for these inputs, or the incompatible component and required versions are clearly identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.