sagemaker: Support multi-model endpoints
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the feature
As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):
> [Multi-Model Endpoints][multi-model]: By default (and as [described in the technical solution above][model-data]), SageMaker expects the model data URL on each container to point to an S3 object containing a gzipped tar file of artifacts, which will be automatically extracted upon instance provisioning. To support colocation of multiple logical models into a single container, the `Mode` attribute was added to the `ContainerDefinition` CloudFormation structure to either explicit configure `SingleModel` mode (the default) or `MultiModel` mode. In multi-model mode, SageMaker now expects the customer configured model data URL to point to an S3 path under which multiple gzipped tar files exist. When invoking a multi-model endpoint, the client invoking the endpoint must specify the target model representing the exact S3 path suffix pointing to a specific gzipped tar file.
[multi-model]: https://aws.amazon.com/blogs/machine-learning/save-on-inference-costs-by-using-amazon-sagemaker-multi-model-endpoints/
[model-data]: https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#model-data
**Please 👍 this issue to help with the prioritization of this feature.**
### Use Case
Colocating multiple models behind a single endpoint can help customers save money as [described here](https://aws.amazon.com/blogs/machine-learning/save-on-inference-costs-by-using-amazon-sagemaker-multi-model-endpoints/).
### Proposed Solution
As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):
> To accommodate this feature, the proposed `ModelData.fromAsset` API should be adjusted to support zip file assets capable of containing one or more gzipped tar files within them. Even though the code need not be aware of `.tar.gz` files specifically, it might prove a better customer experience to at least put up guard rails to prevent zip file assets from being used in single model mode where as multi-model mode could be more permissive.
### Other Information
_No response_
### Acknowledgements
- [ ] I _may_ be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.54.0-alpha.0
### Environment details (OS name and version, etc.)
macOS Ventura
Contributor guide
Research direction
Start with the SageMaker Endpoint L2 construct RFC, especially its feature additions and model-data sections. Then inspect the Endpoint implementation and ModelData.fromAsset API to determine the integration points. Done means the construct supports multi-model endpoints and the proposed asset behavior, with coverage for the supported configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100