apple / apple/coremltools

Flexible size causes error for dynamic resizing

Open
#1,456 1 comment 0 reactions 0 assignees View on GitHub
Flexible Shape PyTorch (traced) question
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## ❓Question
I tried this code,

```python
import pytorch_lightning as pl
import torch
import coremltools
from coremltools.models.neural_network import flexible_shape_utils
from torch import FloatTensor
import torch.nn as nn

class Decoder(pl.LightningModule):
def __init__(self):
super().__init__()
self.word_embed = nn.Sequential(
nn.Embedding(118, 256), nn.LayerNorm(256)
)

def forward(self, src: FloatTensor) -> FloatTensor:
# return torch.flatten(src, 0, 1)
return self.word_embed(src)

model = Decoder()
with torch.no_grad():
input_tensor = torch.ones([1, 10], dtype=torch.int32)
traced_model = torch.jit.trace(model.eval(), input_tensor, check_trace=True, check_tolerance=True)
coreml_model = coremltools.convert(
traced_model,
inputs=[coremltools.TensorType(shape=input_tensor.shape, dtype=int)]
)
coreml_model.save("decoder.mlmodel")

spec = coremltools.utils.load_spec('decoder.mlmodel')
input_name = spec.description.input[0].name
flexible_shape_utils.set_multiarray_ndshape_range(spec,
feature_name=input_name,
lower_bounds=[1, 5],
upper_bounds=[1, 10])
coremltools.utils.save_spec(spec, 'decoder.mlmodel')

loaded = coremltools.models.MLModel('decoder.mlmodel')
input_tensor = input_tensor.detach().numpy().copy()
out = loaded.predict({'src': input_tensor}) # this is OK
print(out)
input_tensor = torch.ones([1, 5], dtype=torch.int32).detach().numpy().copy()
out = loaded.predict({'src': input_tensor}) # error occured
print(out)
```

I got error.
```
Traceback (most recent call last):
File "/Users/ryosukefukatani/work/HMERModel/atnBTTR/d2.py", line 42, in
out = loaded.predict({'src': input_tensor})
File "/Users/ryosukefukatani/work/HMERModel/venv/lib/python3.9/site-packages/coremltools/models/model.py", line 512, in predict
return self.__proxy__.predict(data, useCPUOnly)
RuntimeError: {
NSLocalizedDescription = "Failure dynamically resizing for sequence length.";
}
```
When I removed `nn.LayerNorm(256)`, this code was passed.
How to fix it? or `LayerNorm` does not support flexible size?

### Environment
- macOS 12.1
- coremltools 5.2
- torch 1.10.2
- Python 3.9.6

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied Python example with nn.LayerNorm, coremltools.convert, and flexible_shape_utils.set_multiarray_ndshape_range using input shapes [1, 10] and [1, 5]. Investigate the dynamic-resizing path involved in the LayerNorm-converted model; done means the shorter prediction succeeds or the limitation is clearly identified and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.