aws / aws/aws-cdk

sagemaker: Support multi-model endpoints

Open
#23,154 0 comments 3 reactions 0 assignees View on GitHub
@aws-cdk/aws-sagemaker effort/medium feature-request p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the feature

As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):

> [Multi-Model Endpoints][multi-model]: By default (and as [described in the technical solution above][model-data]), SageMaker expects the model data URL on each container to point to an S3 object containing a gzipped tar file of artifacts, which will be automatically extracted upon instance provisioning. To support colocation of multiple logical models into a single container, the `Mode` attribute was added to the `ContainerDefinition` CloudFormation structure to either explicit configure `SingleModel` mode (the default) or `MultiModel` mode. In multi-model mode, SageMaker now expects the customer configured model data URL to point to an S3 path under which multiple gzipped tar files exist. When invoking a multi-model endpoint, the client invoking the endpoint must specify the target model representing the exact S3 path suffix pointing to a specific gzipped tar file.

[multi-model]: https://aws.amazon.com/blogs/machine-learning/save-on-inference-costs-by-using-amazon-sagemaker-multi-model-endpoints/
[model-data]: https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#model-data

**Please 👍 this issue to help with the prioritization of this feature.**

### Use Case

Colocating multiple models behind a single endpoint can help customers save money as [described here](https://aws.amazon.com/blogs/machine-learning/save-on-inference-costs-by-using-amazon-sagemaker-multi-model-endpoints/).

### Proposed Solution

As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):

> To accommodate this feature, the proposed `ModelData.fromAsset` API should be adjusted to support zip file assets capable of containing one or more gzipped tar files within them. Even though the code need not be aware of `.tar.gz` files specifically, it might prove a better customer experience to at least put up guard rails to prevent zip file assets from being used in single model mode where as multi-model mode could be more permissive.

### Other Information

_No response_

### Acknowledgements

- [ ] I _may_ be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.54.0-alpha.0

### Environment details (OS name and version, etc.)

macOS Ventura

Contributor guide

Open the contributing guide

Research direction

Start with the SageMaker Endpoint L2 construct RFC, especially its feature additions and model-data sections. Then inspect the Endpoint implementation and ModelData.fromAsset API to determine the integration points. Done means the construct supports multi-model endpoints and the proposed asset behavior, with coverage for the supported configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.