aws / aws/sagemaker-pytorch-inference-toolkit

Launch TorchServe without repackaging model contents

Open
#117 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
143
Forks
73
PR merge metrics
No merged PRs in 30d

Description

Currently, the startup code will repackage the model contents in `environment.model_dir` into TS format using the TS model archiver: https://github.com/aws/sagemaker-pytorch-inference-toolkit/blob/master/src/sagemaker_pytorch_serving_container/torchserve.py#L78

This causes the model contents to be read and rewritten to disk on container startup, which increases container startup time. For SageMaker Serverless Inference, this causes cold starts to be longer (even longer for larger models).

TorchServe is making a change to support loading models from a directory without the need to repackage the model as a .mar file: https://github.com/pytorch/serve/issues/1498

This issue is to request for this inference toolkit to use this new feature and avoid repackaging the model contents. I /think/ this should be as simple as removing the [model archiver command execution](https://github.com/aws/sagemaker-pytorch-inference-toolkit/blob/master/src/sagemaker_pytorch_serving_container/torchserve.py#L129) and setting [`--models` in the [TorchServe command](https://github.com/aws/sagemaker-pytorch-inference-toolkit/blob/master/src/sagemaker_pytorch_serving_container/torchserve.py#L94-L95) to `--models model=environment.model_dir`

Contributor guide

Open the contributing guide

Research direction

Inspect src/sagemaker_pytorch_serving_container/torchserve.py at lines 78, 94-95, and 129, then check whether the TorchServe directory-loading change from issue 1498 is available for this toolkit's supported version. The work is done when startup launches the model from environment.model_dir without repackaging its contents, while serving still starts successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.