apple / apple/coremltools

Error during scripted pytorch model to coreml

Open
#1,728 2 comments 0 reactions 0 assignees View on GitHub
PyTorch (not traced) question
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## ❓Question

While converting pytroch model to coreml, i've got error below. I know converting scripted model is experimental, but converting traced model takes so~~~~ long in running MIL common pass step. after long time, it fails with some error saying leaked semaphore. maybe due to small memory?

So, i tried converting scripted model. but also fails with error below. how to fix this error?

UPD

converting traced model stuck at here, and never proceeds.

```
Converting PyTorch Frontend ==> MIL Ops: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 46688/46689 [04:44<00:00, 163.97 ops/s]
Running MIL Common passes: 5%|████████▎ | 2/39 [00:00<00:05, 7.36 passes/s]
```

### Error

```
scikit-learn version 1.2.0 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Using cache found in ./hub/bshall_hubert_main
Using cache found in ./hub/bshall_acoustic-model_main
Support for converting Torch Script Models is experimental. If possible you should use a traced model for conversion.
Converting PyTorch Frontend ==> MIL Ops: 25%|████████████████████████████████████▌ | 28/112 [00:00<00:00, 14386.93 ops/s]
Traceback (most recent call last):
File "/Users/seastar105/Work/soft-vc-ct/acoustic_conversion.py", line 31, in
mlprogram = ct.convert(
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/_converters_entry.py", line 444, in convert
mlmodel = mil_convert(
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 190, in mil_convert
return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 217, in _mil_convert
proto, mil_program = mil_convert_to_proto(
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 282, in mil_convert_to_proto
prog = frontend_converter(model, **kwargs)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 112, in __call__
return load(*args, **kwargs)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 57, in load
return _perform_torch_convert(converter, debug)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 96, in _perform_torch_convert
prog = converter.convert()
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 270, in convert
convert_nodes(self.context, self.graph)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 103, in convert_nodes
add_op(context, node)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 4136, in noop
inputs = _get_inputs(context, node)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 200, in _get_inputs
inputs = [context[name] for name in node.inputs]
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 200, in
inputs = [context[name] for name in node.inputs]
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 78, in __getitem__
raise ValueError(
ValueError: Torch var training.8 not found in context
```

### Environment
coremltools version: 6.1
OS (e.g. MacOS version or Linux type): 13.1 Ventura
Any other relevant version information (e.g. PyTorch or TensorFlow version): Pytorch 1.12.1
`conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 -c pytorch`
`pip install coremltools scikit-learn matplotlib`

### Code for reproduce

```
import torch
import torchaudio
import coremltools as ct

class Acoustic(torch.nn.Module):
def __init__(self, acoustic):
super(Acoustic, self).__init__()
self.acoustic = acoustic

def forward(self, units: torch.Tensor) -> torch.Tensor:
mels = self.acoustic.generate(units).transpose(1, 2)
return mels

torch.hub.set_dir('./hub')
hubert = torch.hub.load("bshall/hubert:main", "hubert_soft").cpu()
acoustic = torch.hub.load("bshall/acoustic-model:main", "hubert_soft").cpu()
model = Acoustic(acoustic).eval()

source, sr = torchaudio.load("source.wav")
source = torchaudio.functional.resample(source, sr, 16000)
source = source.unsqueeze(0)
source = hubert.units(source)

with torch.inference_mode():
origin_shape = model(source).shape
with torch.jit.optimized_execution(True):
scripted = torch.jit.script(model)
scripted_shape = scripted(source).shape
mlprogram = ct.convert(
scripted,
convert_to="mlprogram",
inputs=[ct.TensorType(name="source", shape=source.shape)],
compute_units=ct.ComputeUnit.ALL,
compute_precision=ct.precision.FLOAT16,
)
print("Conversion Success!")
mlprogram.save("acoustic.mlpackage")
print("Save Success!")
```

https://user-images.githubusercontent.com/30820469/211758538-1e219e54-1c11-4c04-b2ab-b095425c2907.mp4

download `mp4` above, and change name to `source.wav` to execute.

after execute code above once, execution would fail if there's no cuda gpu in your machine.

you should modify function line 128 at `hub/bshall_acoustic-model_main/acoustic/model.py`

` checkpoint = torch.hub.load_state_dict_from_url(URLS[name], progress=progress)`
->
` checkpoint = torch.hub.load_state_dict_from_url(URLS[name], progress=progress, map_location="cpu")`

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction in acoustic_conversion.py and the traceback through coremltools/converters/mil/frontend/torch, then inspect hub/bshall_acoustic-model_main/acoustic/model.py around line 128 for the CPU loading adjustment. Run the provided script with source.wav and compare scripted and traced conversion behavior; done means the conversion path is identified and the reported Torch var error or conversion failure is reproducible and explained.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.