aws / aws/sagemaker-pytorch-inference-toolkit

Serving a model using custom container, instance run of disk

Open
#112 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
143
Forks
73
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
Using a custom container to serve a Pytorch model, defined as below, it throw "No space left on device"

```
container = {"Image": image, "ModelDataUrl": model_artifact}

create_model_response = sm.create_model(
ModelName=model_name, ExecutionRoleArn=role, PrimaryContainer=container
)

create_endpoint_config_response = sm.create_endpoint_config(
EndpointConfigName=endpoint_config_name,
ProductionVariants=[
{
"InstanceType": "ml.g4dn.8xlarge",
"InitialVariantWeight": 1,
"InitialInstanceCount": 1,
"ModelName": model_name,
"VariantName": "AllTraffic",
}
],
)

```
Docker image size is 17 GB and Torchserve mar file is 8 GB. I was wondering if there is any way to increase the storage for the instances that are serving the model. Going through the [doc](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html) for endpoint configuration seems there is no setting for specifics about instances.

-- Cloud watch log

![256717956_890382124957120_3900367258239977898_n](https://user-images.githubusercontent.com/9162336/142566990-ae527283-b6da-42dc-aa49-3e2efbea049f.png)

## Expected behavior

Having knobs to set the storage for the serving instances.

Contributor guide

Open the contributing guide

Research direction

The report uses SageMaker create_model and create_endpoint_config with a custom container, and includes a CloudWatch log showing the disk-space failure. Start with the linked CreateEndpointConfig documentation and the reported image and model sizes; determine whether serving-instance storage is configurable and whether that behavior belongs in this toolkit. Done means the supported configuration or limitation is established and reflected in an appropriate change or response.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.