microsoft / microsoft/onnxruntime

Cannot use --use_external_data_form and --optimize_onnx with GPT2LMHeadModel_NoPadding

Open
#6,047 3 comments 1 reaction 1 assignee View on GitHub

@tianleiwu is already working on this.

Since Jan 8, 2021.

model:transformer
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

**Describe the bug**
Exporting a `gpt2` model using the provided `convert_to_onnx` script using `GPT2LMHeadModel_NoPadding` and both `--use_external_data_form` and `--optimize_onnx` flags, results in a model that cannot be loaded successfully. Using `GPT2LMHeadModel` model_class does not produces the same problem.

```
$ python -m onnxruntime_tools.transformers.convert_to_onnx -m gpt2 --model_class GPT2LMHeadModel_NoPadding --output ~/models/gpt2_optimized -p fp32 --use_gpu --use_external_data_form --optimize_onnx
Arguments:Namespace(cache_dir='./cache_models', input_test_file='', model_class='GPT2LMHeadModel_NoPadding', model_name_or_path='gpt2', optimize_onnx=True, output='/home/ec2-user/models/gpt2_optimized', precision=, tolerance=0.0005, use_external_data_format=True, use_gpu=True, verbose=False)
PyTorch Version:1.7.0
Transformers Version:3.1.0
Onnxruntime Version:1.5.2
Exporting ONNX model to /home/ec2-user/models/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past/gpt2_GPT2LMHeadModel_NoPadding_past.onnx
/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/transformers/modeling_gpt2.py:712: FutureWarning: The `past` argument is deprecated and will be removed in a future version, use `past_key_values` instead.
FutureWarning,
Shapes: input_ids=torch.Size([1, 1]) past=torch.Size([2, 1, 12, 1, 64]) output=torch.Size([1, 1, 50257]) present=torch.Size([2, 1, 12, 2, 64])
/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/transformers/modeling_gpt2.py:165: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
w = w / (float(v.size(-1)) ** 0.5)
/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/transformers/modeling_gpt2.py:170: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
mask = self.bias[:, :, ns - nd : ns, :ns]
Optimizing model to /home/ec2-user/models/conv/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past_fp32/gpt2_GPT2LMHeadModel_NoPadding_past_fp32.onnx
Fused LayerNormalization count: 25
Fused FastGelu count: 12
Fused Attention(with past) count: 12
Graph pruned: 0 inputs, 0 outputs and 720 nodes are removed
Graph pruned: 0 inputs, 0 outputs and 312 nodes are removed
postprocess: remove Reshape count:48
Fused FastGelu(add bias) count: 12
opset verion: 11
Output model to /home/ec2-user/models/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past_fp32/gpt2_GPT2LMHeadModel_NoPadding_past_fp32.onnx
Output path: /home/ec2-user/models/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past_fp32/gpt2_GPT2LMHeadModel_NoPadding_past_fp32.onnx
Exception
Traceback (most recent call last):
File "/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/onnxruntime_tools/transformers/benchmark_helper.py", line 59, in create_onnxruntime_session
session = InferenceSession(onnx_model_path, sess_options, providers=execution_providers)
File "/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/onnxruntime/capi/session.py", line 195, in __init__
self._create_inference_session(providers, provider_options)
File "/home/ec2-user/miniconda/envs/conv-onnx/lib/python3.6/site-packages/onnxruntime/capi/session.py", line 200, in _create_inference_session
sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from /home/ec2-user/models/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past_fp32/gpt2_GPT2LMHeadModel_NoPadding_past_fp32.onnx failed:Node (Squeeze_16) Op (Squeeze) [ShapeInferenceError] Dimension of input 0 must be 1 instead of 4
Done. Output model: /home/ec2-user/models/gpt2_optimized/gpt2_GPT2LMHeadModel_NoPadding_past_fp32/gpt2_GPT2LMHeadModel_NoPadding_past_fp32.onnx
```

Running the same command without `--optimize_onnx` or without `--use_external_data_form` works fine. Using `GPT2LMHeadModel` instead of `GPT2LMHeadModel_NoPadding` also fixes the problem.

**Urgency**
There are no hard deadlines. Using `GPT2LMHeadModel` instead of `GPT2LMHeadModel_NoPadding` is an option.

**System information**
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Amazon Linux 2
- ONNX Runtime installed from (source or binary): binary
- ONNX Runtime version: 1.5.2
- Python version: 3.6

**To Reproduce**

```
python -m onnxruntime_tools.transformers.convert_to_onnx -m gpt2 --model_class GPT2LMHeadModel_NoPadding --output ~/models/gpt2_optimized -p fp32 --use_gpu --use_external_data_form --optimize_onnx
```

**Expected behavior**
Using `--use_external_data_form` and `--optimize_onnx` with model class `GPT2LMHeadModel_NoPadding` works.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.