Error during scripted pytorch model to coreml
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
## ❓Question
While converting pytroch model to coreml, i've got error below. I know converting scripted model is experimental, but converting traced model takes so~~~~ long in running MIL common pass step. after long time, it fails with some error saying leaked semaphore. maybe due to small memory?
So, i tried converting scripted model. but also fails with error below. how to fix this error?
UPD
converting traced model stuck at here, and never proceeds.
```
Converting PyTorch Frontend ==> MIL Ops: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 46688/46689 [04:44<00:00, 163.97 ops/s]
Running MIL Common passes: 5%|████████▎ | 2/39 [00:00<00:05, 7.36 passes/s]
```
### Error
```
scikit-learn version 1.2.0 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Using cache found in ./hub/bshall_hubert_main
Using cache found in ./hub/bshall_acoustic-model_main
Support for converting Torch Script Models is experimental. If possible you should use a traced model for conversion.
Converting PyTorch Frontend ==> MIL Ops: 25%|████████████████████████████████████▌ | 28/112 [00:00<00:00, 14386.93 ops/s]
Traceback (most recent call last):
File "/Users/seastar105/Work/soft-vc-ct/acoustic_conversion.py", line 31, in
mlprogram = ct.convert(
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/_converters_entry.py", line 444, in convert
mlmodel = mil_convert(
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 190, in mil_convert
return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 217, in _mil_convert
proto, mil_program = mil_convert_to_proto(
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 282, in mil_convert_to_proto
prog = frontend_converter(model, **kwargs)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 112, in __call__
return load(*args, **kwargs)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 57, in load
return _perform_torch_convert(converter, debug)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 96, in _perform_torch_convert
prog = converter.convert()
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 270, in convert
convert_nodes(self.context, self.graph)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 103, in convert_nodes
add_op(context, node)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 4136, in noop
inputs = _get_inputs(context, node)
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 200, in _get_inputs
inputs = [context[name] for name in node.inputs]
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 200, in
inputs = [context[name] for name in node.inputs]
File "/Users/seastar105/opt/anaconda3/envs/soft-vc-ct/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 78, in __getitem__
raise ValueError(
ValueError: Torch var training.8 not found in context
```
### Environment
coremltools version: 6.1
OS (e.g. MacOS version or Linux type): 13.1 Ventura
Any other relevant version information (e.g. PyTorch or TensorFlow version): Pytorch 1.12.1
`conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 -c pytorch`
`pip install coremltools scikit-learn matplotlib`
### Code for reproduce
```
import torch
import torchaudio
import coremltools as ct
class Acoustic(torch.nn.Module):
def __init__(self, acoustic):
super(Acoustic, self).__init__()
self.acoustic = acoustic
def forward(self, units: torch.Tensor) -> torch.Tensor:
mels = self.acoustic.generate(units).transpose(1, 2)
return mels
torch.hub.set_dir('./hub')
hubert = torch.hub.load("bshall/hubert:main", "hubert_soft").cpu()
acoustic = torch.hub.load("bshall/acoustic-model:main", "hubert_soft").cpu()
model = Acoustic(acoustic).eval()
source, sr = torchaudio.load("source.wav")
source = torchaudio.functional.resample(source, sr, 16000)
source = source.unsqueeze(0)
source = hubert.units(source)
with torch.inference_mode():
origin_shape = model(source).shape
with torch.jit.optimized_execution(True):
scripted = torch.jit.script(model)
scripted_shape = scripted(source).shape
mlprogram = ct.convert(
scripted,
convert_to="mlprogram",
inputs=[ct.TensorType(name="source", shape=source.shape)],
compute_units=ct.ComputeUnit.ALL,
compute_precision=ct.precision.FLOAT16,
)
print("Conversion Success!")
mlprogram.save("acoustic.mlpackage")
print("Save Success!")
```
https://user-images.githubusercontent.com/30820469/211758538-1e219e54-1c11-4c04-b2ab-b095425c2907.mp4
download `mp4` above, and change name to `source.wav` to execute.
after execute code above once, execution would fail if there's no cuda gpu in your machine.
you should modify function line 128 at `hub/bshall_acoustic-model_main/acoustic/model.py`
` checkpoint = torch.hub.load_state_dict_from_url(URLS[name], progress=progress)`
->
` checkpoint = torch.hub.load_state_dict_from_url(URLS[name], progress=progress, map_location="cpu")`
Contributor guide
Research direction
Start with the reproduction in acoustic_conversion.py and the traceback through coremltools/converters/mil/frontend/torch, then inspect hub/bshall_acoustic-model_main/acoustic/model.py around line 128 for the CPU loading adjustment. Run the provided script with source.wav and compare scripted and traced conversion behavior; done means the conversion path is identified and the reported Torch var error or conversion failure is reproducible and explained.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100