aws / aws/sagemaker-pytorch-inference-toolkit
Launch TorchServe without repackaging model contents
- Dominant language
- Python
- Stars
- 143
- Forks
- 73
- PR merge metrics
- No merged PRs in 30d
Description
Currently, the startup code will repackage the model contents in `environment.model_dir` into TS format using the TS model archiver: https://github.com/aws/sagemaker-pytorch-inference-toolkit/blob/master/src/sagemaker_pytorch_serving_container/torchserve.py#L78
This causes the model contents to be read and rewritten to disk on container startup, which increases container startup time. For SageMaker Serverless Inference, this causes cold starts to be longer (even longer for larger models).
TorchServe is making a change to support loading models from a directory without the need to repackage the model as a .mar file: https://github.com/pytorch/serve/issues/1498
This issue is to request for this inference toolkit to use this new feature and avoid repackaging the model contents. I /think/ this should be as simple as removing the [model archiver command execution](https://github.com/aws/sagemaker-pytorch-inference-toolkit/blob/master/src/sagemaker_pytorch_serving_container/torchserve.py#L129) and setting [`--models` in the [TorchServe command](https://github.com/aws/sagemaker-pytorch-inference-toolkit/blob/master/src/sagemaker_pytorch_serving_container/torchserve.py#L94-L95) to `--models model=environment.model_dir`
Contributor guide
Research direction
Inspect src/sagemaker_pytorch_serving_container/torchserve.py at lines 78, 94-95, and 129, then check whether the TorchServe directory-loading change from issue 1498 is available for this toolkit's supported version. The work is done when startup launches the model from environment.model_dir without repackaging its contents, while serving still starts successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100