aws / aws/amazon-sagemaker-examples
[Bug Report]
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
[All the PyTorch NEO compilation jobs in this directory](https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker_neo_compilation_jobs)
**Describe the bug**
Running the pytorch example notebooks:
- [inf1_bert_compile_and_deploy](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker_neo_compilation_jobs/deploy_huggingface_model_on_Inf1_instance/inf1_bert_compile_and_deploy.ipynb)
- [pytorch_torchvision_neo_on_Inf1](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker_neo_compilation_jobs/deploy_pytorch_model_on_Inf1_instance/pytorch_torchvision_neo_on_Inf1.ipynb)
- [pytorch_torchvision_neo](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker_neo_compilation_jobs/pytorch_torchvision/pytorch_torchvision_neo.ipynb)
unchanged, on ml.c5.xlarge, conda_pytorch_p38 kernel, yields the following error:
```
ClientError: An error occurred (ValidationException) when calling the CreateCompilationJob operation: Unsupported framework version field for target. Framework version is supported for Target Platform configuration and only part of target devices.
Framework version is only supported for ml_c4, ml_c5, ml_m4, ml_m5, ml_p2, ml_p3, ml_g4dn cloud targetsand lambda, jetson_tx1, jetson_nano, jetson_tx2, jetson_xavier, deeplens, rasp3b, rasp4b, imx8qm, rk3288, rk3399, aisage, sbe_c, qcs605, qcs603, x86_win32, x86_win64 edge devices.
```
The same run was working on Friday morning.
**To reproduce**
- Launch a sagemaker notebook instance on a ml.c5.xlarge machine, tweaking memory to 15GB and cloning the `https://github.com/aws/amazon-sagemaker-examples/` repo at in startup configuration
- Upon startup, launch one of the pytorch neo compilation notebooks (listed above) with the conda_pytorch_p38 kernel and execute in order the notebook cells.
- The ClientError occurs at the compilation step (looks like `neo_model = pytorch_model.compile(...)`)
**What I've tried**
- I've tried changing the notebook pytorch version to every version between 1.5.1 and 1.11.0
- I've tried changing the framework_version argument of the `PyTorchModel` object and the `compile` method to the corresponding versions.
I would appreciate any help in sorting out what is going wrong.
Dan Ringwald
Contributor guide
Assessment
This issue has not been assessed yet.