alibaba / alibaba/TinyNeuralNetwork

LayerNorm conversion error

Open
#393 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
879
Forks
134
PR merge metrics
No merged PRs in 30d

Description

Hi, In the latest version of TinyNeuralNetwork layer norm causes the conversion to fail.

error output:
```
Error in QNNPACK: failed to create add operator with 8.124962e-06 A-to-output scale ratio: scale ratio must be in [2**-14, 2**8) range
...
File "../TinyNeuralNetwork/tinynn/graph/quantization/modules.py", line 136, in forward
return self.f_add_2.add(norm_alpha, bias_fq_expand)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "../torch/ao/nn/quantized/modules/functional_modules.py", line 241, in add
r = ops.quantized.add(x, y, scale=self.scale, zero_point=self.zero_point)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "../torch/_ops.py", line 1116, in __call__
return self._op(*args, **(kwargs or {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: createStatus == pytorch_qnnp_status_success INTERNAL ASSERT FAILED at "../aten/src/ATen/native/quantized/cpu/BinaryOps.cpp":204, please report a bug to PyTorch. failed to create QNNPACK Add operator
```

torch version: 2.5.1
python version: 3.12

This should reproduce it:
```python
import torch.nn as nn
import torch
from tinynn.graph.quantization.quantizer import PostQuantizer
from tinynn.converter import TFLiteConverter
from tinynn.graph.tracer import model_tracer

class LayerNormModel(nn.Module):

def __init__(self,):
super().__init__()
self.layer_norm = torch.nn.LayerNorm(256)

def forward(self, x: torch.Tensor):
return self.layer_norm(x)

def _main():
dummy_input = torch.rand(1, 60, 256).float()
model = LayerNormModel()
qat_config = {
"backend": "qnnpack",
"per_tensor": True,
"disable_requantization_for_cat": True,
}
with model_tracer():
quantizer = PostQuantizer(
model, (dummy_input), work_dir="LayerNormModel", config=qat_config
)

layer_norm_model = quantizer.quantize()

layer_norm_model(dummy_input)

with torch.no_grad():
layer_norm_model.eval()
layer_norm_model.cpu()

layer_norm_model = quantizer.convert(layer_norm_model)
torch.backends.quantized.engine = quantizer.backend
converter = TFLiteConverter(
layer_norm_model,
(dummy_input),
"layer_norm.tflite",
fuse_quant_dequant=True,
quantize_target_type="int8"
)
converter.convert()

if __name__ == '__main__':
_main()
```

Is there a new flag or something I should set to make this work?

Contributor guide

Open the contributing guide

Research direction

Run the supplied LayerNormModel reproduction with torch 2.5.1 and Python 3.12, then inspect tinynn/graph/quantization/modules.py at the reported forward call and the PostQuantizer conversion path. Trace the QNNPACK scale-ratio failure through quantization and conversion. Done means the example converts to layer_norm.tflite without the QNNPACK error, with coverage for this LayerNorm case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.