aws / aws/amazon-sagemaker-examples

[Bug Report]

Open
#3,239 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
[All the PyTorch NEO compilation jobs in this directory](https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker_neo_compilation_jobs)

**Describe the bug**

Running the pytorch example notebooks:
- [inf1_bert_compile_and_deploy](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker_neo_compilation_jobs/deploy_huggingface_model_on_Inf1_instance/inf1_bert_compile_and_deploy.ipynb)
- [pytorch_torchvision_neo_on_Inf1](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker_neo_compilation_jobs/deploy_pytorch_model_on_Inf1_instance/pytorch_torchvision_neo_on_Inf1.ipynb)
- [pytorch_torchvision_neo](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker_neo_compilation_jobs/pytorch_torchvision/pytorch_torchvision_neo.ipynb)

unchanged, on ml.c5.xlarge, conda_pytorch_p38 kernel, yields the following error:

```
ClientError: An error occurred (ValidationException) when calling the CreateCompilationJob operation: Unsupported framework version field for target. Framework version is supported for Target Platform configuration and only part of target devices.
Framework version is only supported for ml_c4, ml_c5, ml_m4, ml_m5, ml_p2, ml_p3, ml_g4dn cloud targetsand lambda, jetson_tx1, jetson_nano, jetson_tx2, jetson_xavier, deeplens, rasp3b, rasp4b, imx8qm, rk3288, rk3399, aisage, sbe_c, qcs605, qcs603, x86_win32, x86_win64 edge devices.
```

The same run was working on Friday morning.

**To reproduce**

- Launch a sagemaker notebook instance on a ml.c5.xlarge machine, tweaking memory to 15GB and cloning the `https://github.com/aws/amazon-sagemaker-examples/` repo at in startup configuration
- Upon startup, launch one of the pytorch neo compilation notebooks (listed above) with the conda_pytorch_p38 kernel and execute in order the notebook cells.
- The ClientError occurs at the compilation step (looks like `neo_model = pytorch_model.compile(...)`)

**What I've tried**

- I've tried changing the notebook pytorch version to every version between 1.5.1 and 1.11.0
- I've tried changing the framework_version argument of the `PyTorchModel` object and the `compile` method to the corresponding versions.

I would appreciate any help in sorting out what is going wrong.

Dan Ringwald

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.