aws / aws/amazon-sagemaker-examples

YoloV5s SageMaker Neo compilation failed

Open
#2,784 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

https://github.com/aws-samples/aws-panorama-samples/blob/main/Object-Detection-YOLOv5/yolov5s.ipynb

I followed the code in the linked Notebook on how to run a **YoloV5 model** on AWS Panorama. In order to do that I first need to get a pretrained (on COCO dataset) YoloV5s model (from this repository https://github.com/ultralytics/yolov5). Then I exported it as a tar file and uploaded to an S3 bucket. This all worked fine, I followed the code from the Notebook and ran it on a SageMaker Instance using conda_python3 kernel.

Once the model was uploaded to S3 I started a **Compilation Job in SageMaker Neo** from the console with the following parameters:

- Location of model artifacts: the S3 bucket URL with the model tar file
- Data input configuration: {"data":[1, 3, 640, 640]}
- Machine learning framework: PYTORCH
- Target device: jetson_xavier
- Output location: an S3 bucket

The Job failes after a few minutes with the following error:
`ClientError: InputConfiguration: TVM cannot convert the PyTorch model. Invalid model or input-shape mismatch. Make sure that inputs are lexically ordered and of the correct dimensionality. Traceback (most recent call last):\n [bt] (4) /opt/amazon/lib/python3.6/site-packages/tvm/libtvm.so(TVMFuncCall+0x61) [0x7f1661918f41]\n [bt] (3) /opt/amazon/lib/python3.6/site-packages/tvm/libtvm.so(+0x1ada205) [0x7f16617da205]\n [bt] (2) /opt/amazon/lib/python3.6/site-packages/tvm/libtvm.so(tvm::runtime::TVMMovableArgValueWithContext_::operator tvm::runtime::Array >() const+0x5e) [0x7f16617e059e]\n [bt] (1) /opt/amazon/lib/python3.6/site-packages/tvm/libtvm.so(tvm::runtime::Array tvm::runtime::TVMPODValue_::AsObjectRef >() const+0x413) [0x7f16612e8853]\n [bt] (0) /opt/amazon/lib/python3.6/site-packages/tvm/libtvm.so(+0x15ddf7f) [0x7f16612ddf7f]\n File "/tvm/include/tvm/runtime/packed_func.h", line 687\nTVMError: In function relay.ir.Tuple: error while converting argument 0: [11:54:33] /tvm/include/tvm/runtime/packed_func.h:1564: \n---------------------------------------------------------------\nAn internal invariant was violated during the execution of TVM.\nPlease read TVM\'s error reporting guidelines.\nMore details can be found here: https://discuss.tvm.ai/t/error-reporting/7793.\n---------------------------------------------------------------\n Check failed: !checked_type.defined() == false: Expected Array[RelayExpr], but got Array[index 1: Array]\n`

Screenshot 2021-06-30 at 09 23 27

I also tried by using different custom trained YoloV5s model. I prepared the models as described in this link: https://docs.aws.amazon.com/sagemaker/latest/dg/neo-compilation-preparing-model.html.
Again, I get the same error. With another model I got this error: `Failure reason
ClientError: InputConfiguration: TVM cannot convert the PyTorch model. Invalid model or input-shape mismatch. Make sure that inputs are lexically ordered and of the correct dimensionality. The following operators are not implemented: ['aten::silu_', 'aten::copy_']`

Contributor guide

Open the contributing guide

Research direction

Start with Object-Detection-YOLOv5/yolov5s.ipynb and reproduce the SageMaker Neo compilation using the linked model artifacts and input configuration. Compare the reported TVM conversion failures, including the unsupported operators, with the notebook's documented workflow. Done is not defined by the issue, so a maintainer would need to specify the expected correction or documentation change.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.