aws / aws/sagemaker-pytorch-inference-toolkit
Serving a model using custom container, instance run of disk
- Dominant language
- Python
- Stars
- 143
- Forks
- 73
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Using a custom container to serve a Pytorch model, defined as below, it throw "No space left on device"
```
container = {"Image": image, "ModelDataUrl": model_artifact}
create_model_response = sm.create_model(
ModelName=model_name, ExecutionRoleArn=role, PrimaryContainer=container
)
create_endpoint_config_response = sm.create_endpoint_config(
EndpointConfigName=endpoint_config_name,
ProductionVariants=[
{
"InstanceType": "ml.g4dn.8xlarge",
"InitialVariantWeight": 1,
"InitialInstanceCount": 1,
"ModelName": model_name,
"VariantName": "AllTraffic",
}
],
)
```
Docker image size is 17 GB and Torchserve mar file is 8 GB. I was wondering if there is any way to increase the storage for the instances that are serving the model. Going through the [doc](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html) for endpoint configuration seems there is no setting for specifics about instances.
-- Cloud watch log

## Expected behavior
Having knobs to set the storage for the serving instances.
Contributor guide
Research direction
The report uses SageMaker create_model and create_endpoint_config with a custom container, and includes a CloudWatch log showing the disk-space failure. Start with the linked CreateEndpointConfig documentation and the reported image and model sizes; determine whether serving-instance storage is configurable and whether that behavior belongs in this toolkit. Done means the supported configuration or limitation is established and reflected in an appropriate change or response.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python, pytorch
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100