aws / aws/amazon-sagemaker-examples

ONNX inference with EI example does not actually use EI

Open
#3,025 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**

https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/mxnet_onnx_eia/mxnet_onnx_eia.ipynb
Especially:
https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/mxnet_onnx_eia/resnet152.py

**What aspects of the notebook can be improved?**
EI is not used during inference (prediction time is same whether you attach EI or not).

**What are your suggestions?**

Replace:
```
mod = mx.mod.Module(symbol=sym, data_names=["data"], label_names=None)
```

With (inspired by https://github.com/aws/sagemaker-mxnet-inference-toolkit/blob/master/src/sagemaker_mxnet_serving_container/default_inference_handler.py#L72):
```
INFERENCE_ACCELERATOR_PRESENT_ENV = 'SAGEMAKER_INFERENCE_ACCELERATOR_PRESENT'
context = mx.cpu()

if os.environ.get(INFERENCE_ACCELERATOR_PRESENT_ENV) == 'true':
context = mx.eia()

mod = mx.mod.Module(symbol=sym, data_names=["data"], label_names=None, context=context)
```

Contributor guide

Open the contributing guide

Research direction

Open sagemaker-python-sdk/mxnet_onnx_eia/mxnet_onnx_eia.ipynb and especially resnet152.py; start by checking how the MXNet Module is initialized for inference. Verify the example selects the inference accelerator when the documented environment variable is present, and confirm the inference path uses EI rather than showing the same prediction time as CPU-only execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.