microsoft / microsoft/onnxruntime
Cannot use --use_external_data_form and --optimize_onnx with GPT2LMHeadModel_NoPadding
@tianleiwu is already working on this.
Since Jan 8, 2021.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
**Describe the bug**
Exporting a `gpt2` model using the provided `convert_to_onnx` script using `GPT2LMHeadModel_NoPadding` and both `--use_external_data_form` and `--optimize_onnx` flags, results in a model that cannot be loaded successfully. Using `GPT2LMHeadModel` model_class does not produces the same problem.
```
$ python -m onnxruntime_tools.transformers.convert_to_onnx -m gpt2 --model_class GPT2LMHeadModel_NoPadding --output ~/models/gpt2_optimized -p fp32 --use_gpu --use_external_data_form --optimize_onnx
Arguments:Namespace(cache_dir='./cache_models', input_test_file='', model_class='GPT2LMHeadModel_NoPadding', model_name_or_path='gpt2', optimize_onnx=True, output='/home/ec2-user/models/gpt2_optimized', precision=, tolerance=0.0005, use_external_data_format=True, use_gpu=True, verbose=False)
PyTorch Version:1.7.0
Transformers Version:3.1.0
Onnxruntime Version:1.5.2
Exporting ONNX model to /home/ec2-user/models/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past/gpt2_GPT2LMHeadModel_NoPadding_past.onnx
/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/transformers/modeling_gpt2.py:712: FutureWarning: The `past` argument is deprecated and will be removed in a future version, use `past_key_values` instead.
FutureWarning,
Shapes: input_ids=torch.Size([1, 1]) past=torch.Size([2, 1, 12, 1, 64]) output=torch.Size([1, 1, 50257]) present=torch.Size([2, 1, 12, 2, 64])
/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/transformers/modeling_gpt2.py:165: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
w = w / (float(v.size(-1)) ** 0.5)
/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/transformers/modeling_gpt2.py:170: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
mask = self.bias[:, :, ns - nd : ns, :ns]
Optimizing model to /home/ec2-user/models/conv/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past_fp32/gpt2_GPT2LMHeadModel_NoPadding_past_fp32.onnx
Fused LayerNormalization count: 25
Fused FastGelu count: 12
Fused Attention(with past) count: 12
Graph pruned: 0 inputs, 0 outputs and 720 nodes are removed
Graph pruned: 0 inputs, 0 outputs and 312 nodes are removed
postprocess: remove Reshape count:48
Fused FastGelu(add bias) count: 12
opset verion: 11
Output model to /home/ec2-user/models/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past_fp32/gpt2_GPT2LMHeadModel_NoPadding_past_fp32.onnx
Output path: /home/ec2-user/models/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past_fp32/gpt2_GPT2LMHeadModel_NoPadding_past_fp32.onnx
Exception
Traceback (most recent call last):
File "/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/onnxruntime_tools/transformers/benchmark_helper.py", line 59, in create_onnxruntime_session
session = InferenceSession(onnx_model_path, sess_options, providers=execution_providers)
File "/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/onnxruntime/capi/session.py", line 195, in __init__
self._create_inference_session(providers, provider_options)
File "/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/onnxruntime/capi/session.py", line 200, in _create_inference_session
sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from /home/ec2-user/models/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past_fp32/gpt2_GPT2LMHeadModel_NoPadding_past_fp32.onnx failed:Node (Squeeze_16) Op (Squeeze) [ShapeInferenceError] Dimension of input 0 must be 1 instead of 4
Done. Output model: /home/ec2-user/models/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past_fp32/gpt2_GPT2LMHeadModel_NoPadding_past_fp32.onnx
```
Running the same command without `--optimize_onnx` or without `--use_external_data_form` works fine. Using `GPT2LMHeadModel` instead of `GPT2LMHeadModel_NoPadding` also fixes the problem.
**Urgency**
There are no hard deadlines. Using `GPT2LMHeadModel` instead of `GPT2LMHeadModel_NoPadding` is an option.
**System information**
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Amazon Linux 2
- ONNX Runtime installed from (source or binary): binary
- ONNX Runtime version: 1.5.2
- Python version: 3.6
**To Reproduce**
```
python -m onnxruntime_tools.transformers.convert_to_onnx -m gpt2 --model_class GPT2LMHeadModel_NoPadding --output ~/models/gpt2_optimized -p fp32 --use_gpu --use_external_data_form --optimize_onnx
```
**Expected behavior**
Using `--use_external_data_form` and `--optimize_onnx` with model class `GPT2LMHeadModel_NoPadding` works.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.