Triton on SageMaker - NLP Bert - Failed to parse ONNX model from file: model.onnx

Hi Team,

Greetings!!

I am trying to execute the instructions in the following notebook but am not able to parse the ONNX model. Could you please have a look?

Notebook:
https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-triton/nlp_bert/triton_nlp_bert.ipynb

Description:
When I tried to generate the models using the command below then I got the parse error.

!docker run --gpus=all --rm -it \
-v `pwd`/workspace:/workspace nvcr.io/nvidia/pytorch:21.08-py3 \
/bin/bash generate_models.sh

Logs:
...........................
PyTorch
...........................

NVIDIA Release 21.08 (build 26011915)
PyTorch Version 1.10.0a0+3fd9dcf

Container image Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

Copyright (c) 2014-2021 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015 Google Inc.
Copyright (c) 2015 Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available.
Use 'nvidia-docker run' to start this container; see
https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker .

NOTE: MOFED driver for multi-node communication was not detected.
Multi-node communication performance may be reduced.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for PyTorch. NVIDIA recommends the use of the following flags:
nvidia-docker run --ipc=host ...

Collecting transformers==4.9.1
Downloading transformers-4.9.1-py3-none-any.whl (2.6 MB)
|████████████████████████████████| 2.6 MB 2.2 MB/s
Collecting tokenizers<0.11,>=0.10.1
Downloading tokenizers-0.10.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3 MB)
|████████████████████████████████| 3.3 MB 1.9 MB/s
Requirement already satisfied: sacremoses in /opt/conda/lib/python3.8/site-packages (from transformers==4.9.1) (0.0.45)
Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.8/site-packages (from transformers==4.9.1) (5.4.1)
Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.8/site-packages (from transformers==4.9.1) (2021.8.3)
Requirement already satisfied: filelock in /opt/conda/lib/python3.8/site-packages (from transformers==4.9.1) (3.0.12)
Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.8/site-packages (from transformers==4.9.1) (1.21.2)
Requirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.8/site-packages (from transformers==4.9.1) (4.62.1)
Requirement already satisfied: requests in /opt/conda/lib/python3.8/site-packages (from transformers==4.9.1) (2.26.0)
Collecting huggingface-hub==0.0.12
Downloading huggingface_hub-0.0.12-py3-none-any.whl (37 kB)
Requirement already satisfied: packaging in /opt/conda/lib/python3.8/site-packages (from transformers==4.9.1) (21.0)
Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from huggingface-hub==0.0.12->transformers==4.9.1) (3.10.0.0)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.8/site-packages (from packaging->transformers==4.9.1) (2.4.7)
Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests->transformers==4.9.1) (3.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.8/site-packages (from requests->transformers==4.9.1) (1.26.6)
Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.8/site-packages (from requests->transformers==4.9.1) (2.0.0)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests->transformers==4.9.1) (2021.5.30)
Requirement already satisfied: six in /opt/conda/lib/python3.8/site-packages (from sacremoses->transformers==4.9.1) (1.16.0)
Requirement already satisfied: joblib in /opt/conda/lib/python3.8/site-packages (from sacremoses->transformers==4.9.1) (1.0.1)
Requirement already satisfied: click in /opt/conda/lib/python3.8/site-packages (from sacremoses->transformers==4.9.1) (7.1.2)
Installing collected packages: tokenizers, huggingface-hub, transformers
Successfully installed huggingface-hub-0.0.12 tokenizers-0.10.3 transformers-4.9.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
': [Errno 2] No such file or directory.py
': [Errno 2] No such file or directoryy
&&&& RUNNING TensorRT.trtexec [TensorRT v8001] # trtexec --onnx=model.onnx --saveEngine=model_bs16.plan --minShapes=token_ids:1x128,attn_mask:1x128 --optShapes=token_ids:16x128,attn_mask:16x128 --maxShapes=token_ids:128x128,attn_mask:128x128 --fp16 --verbose --workspace=14000
[08/02/2022-13:46:01] [I] === Model Options ===
[08/02/2022-13:46:01] [I] Format: ONNX
[08/02/2022-13:46:01] [I] Model: model.onnx
[08/02/2022-13:46:01] [I] Output:
[08/02/2022-13:46:01] [I] === Build Options ===
[08/02/2022-13:46:01] [I] Max batch: explicit
[08/02/2022-13:46:01] [I] Workspace: 14000 MiB
[08/02/2022-13:46:01] [I] minTiming: 1
[08/02/2022-13:46:01] [I] avgTiming: 8
[08/02/2022-13:46:01] [I] Precision: FP32+FP16
[08/02/2022-13:46:01] [I] Calibration:
[08/02/2022-13:46:01] [I] Refit: Disabled
[08/02/2022-13:46:01] [I] Sparsity: Disabled
[08/02/2022-13:46:01] [I] Safe mode: Disabled
[08/02/2022-13:46:01] [I] Restricted mode: Disabled
[08/02/2022-13:46:01] [I] Save engine: model_bs16.plan
[08/02/2022-13:46:01] [I] Load engine:
[08/02/2022-13:46:01] [I] NVTX verbosity: 0
[08/02/2022-13:46:01] [I] Tactic sources: Using default tactic sources
[08/02/2022-13:46:01] [I] timingCacheMode: local
[08/02/2022-13:46:01] [I] timingCacheFile:
[08/02/2022-13:46:01] [I] Input(s)s format: fp32:CHW
[08/02/2022-13:46:01] [I] Output(s)s format: fp32:CHW
[08/02/2022-13:46:01] [I] Input build shape: attn_mask=1x128+16x128+128x128
[08/02/2022-13:46:01] [I] Input build shape: token_ids=1x128+16x128+128x128
[08/02/2022-13:46:01] [I] Input calibration shapes: model
[08/02/2022-13:46:01] [I] === System Options ===
[08/02/2022-13:46:01] [I] Device: 0
[08/02/2022-13:46:01] [I] DLACore:
[08/02/2022-13:46:01] [I] Plugins:
[08/02/2022-13:46:01] [I] === Inference Options ===
[08/02/2022-13:46:01] [I] Batch: Explicit
[08/02/2022-13:46:01] [I] Input inference shape: token_ids=16x128
[08/02/2022-13:46:01] [I] Input inference shape: attn_mask=16x128
[08/02/2022-13:46:01] [I] Iterations: 10
[08/02/2022-13:46:01] [I] Duration: 3s (+ 200ms warm up)
[08/02/2022-13:46:01] [I] Sleep time: 0ms
[08/02/2022-13:46:01] [I] Streams: 1
[08/02/2022-13:46:01] [I] ExposeDMA: Disabled
[08/02/2022-13:46:01] [I] Data transfers: Enabled
[08/02/2022-13:46:01] [I] Spin-wait: Disabled
[08/02/2022-13:46:01] [I] Multithreading: Disabled
[08/02/2022-13:46:01] [I] CUDA Graph: Disabled
[08/02/2022-13:46:01] [I] Separate profiling: Disabled
[08/02/2022-13:46:01] [I] Time Deserialize: Disabled
[08/02/2022-13:46:01] [I] Time Refit: Disabled
[08/02/2022-13:46:01] [I] Skip inference: Disabled
[08/02/2022-13:46:01] [I] Inputs:
[08/02/2022-13:46:01] [I] === Reporting Options ===
[08/02/2022-13:46:01] [I] Verbose: Enabled
[08/02/2022-13:46:01] [I] Averages: 10 inferences
[08/02/2022-13:46:01] [I] Percentile: 99
[08/02/2022-13:46:01] [I] Dump refittable layers:Disabled
[08/02/2022-13:46:01] [I] Dump output: Disabled
[08/02/2022-13:46:01] [I] Profile: Disabled
[08/02/2022-13:46:01] [I] Export timing to JSON file:
[08/02/2022-13:46:01] [I] Export output to JSON file:
[08/02/2022-13:46:01] [I] Export profile to JSON file:
[08/02/2022-13:46:01] [I]
[08/02/2022-13:46:01] [I] === Device Information ===
[08/02/2022-13:46:01] [I] Selected Device: NVIDIA GeForce GTX 1650
[08/02/2022-13:46:01] [I] Compute Capability: 7.5
[08/02/2022-13:46:01] [I] SMs: 14
[08/02/2022-13:46:01] [I] Compute Clock Rate: 1.515 GHz
[08/02/2022-13:46:01] [I] Device Global Memory: 4096 MiB
[08/02/2022-13:46:01] [I] Shared Memory per SM: 64 KiB
[08/02/2022-13:46:01] [I] Memory Bus Width: 128 bits (ECC disabled)
[08/02/2022-13:46:01] [I] Memory Clock Rate: 6.001 GHz
[08/02/2022-13:46:01] [I]
[08/02/2022-13:46:01] [I] TensorRT version: 8001
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::Proposal version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::Split version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[08/02/2022-13:46:01] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[08/02/2022-13:46:01] [I] [TRT] [MemUsageChange] Init CUDA: CPU +326, GPU +0, now: CPU 333, GPU 907 (MiB)
[08/02/2022-13:46:01] [I] Start parsing network model
Could not open file model.onnx
Could not open file model.onnx
[E] [08/02/2022-13:46:01] [TRT] ModelImporter.cpp:682: Failed to parse ONNX model from file: model.onnx
[E] Failed to parse onnx file
[E] Parsing model failed
[E] Engine creation failed
[E] Engine set up failed
[08/02/2022-13:46:01] [08/02/2022-13:46:01] [I] Finish parsing network model
[08/02/2022-13:46:01] [08/02/2022-13:46:01] [08/02/2022-13:46:01] &&&& FAILED TensorRT.trtexec [TensorRT v8001] # trtexec --onnx=model.onnx --saveEngine=model_bs16.plan --minShapes=token_ids:1x128,attn_mask:1x128 --optShapes=token_ids:16x128,attn_mask:16x128 --maxShapes=token_ids:128x128,attn_mask:128x128 --fp16 --verbose --workspace=14000

Start with sagemaker-triton/nlp_bert/triton_nlp_bert.ipynb and the generate_models.sh command shown in the report. Reproduce the model-generation step, verify whether model.onnx is created in the mounted workspace, and confirm that the resulting file can be parsed by the displayed TensorRT command.

Written by the indexing model from the issue text.