apple / apple/coremltools

A model that concats two inputs is not converted correctly.

Open
#1,725 1 comment 0 reactions 0 assignees View on GitHub
awaiting response question
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## ❓Question

I want to convert an inpainting model with two inputs.

https://github.com/researchmm/AOT-GAN-for-Inpainting

One input is the masked original image (x) and the other input is the mask image (mask).
When converted to torchscript, the model concats the two inputs as intended, as shown in the attached image.

```
x = torch.rand(1, 3, 512, 512)
mask = torch.rand(1, 1, 512, 512)

ts = torch.jit.trace(model, (x,mask))
```

スクリーンショット 2023-01-10 18 10 32

However, when I convert to coreml model with the code below, the conversion itself completes correctly, but the converted mlmodel double joins one input (mask) and ignores the other input (x).

```
import coremltools as ct
mlmodel = ct.convert(ts, inputs=[ct.ImageType(shape=x.shape),ct.ImageType(shape=mask.shape)])
mlmodel.save("inpainting.mlmodel")
```

スクリーンショット 2023-01-10 18 11 49

concat receives x:1 and mask.
The output of the ignored input (x) is x:0, and the mask input (mask) outputs x:1 and mask and is concat.

Is there a way to concat the two inputs as intended by the original model?

Thank you for your work. Best,

Contributor guide

Open the contributing guide

Research direction

Start with the Python conversion call using the traced TorchScript model and the two ImageType inputs, then inspect how the converted model represents the concat inputs. Reproduce the AOT-GAN inpainting case with x and mask; done means both inputs feed the concat instead of x being ignored or mask being duplicated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.