sagemaker: Support direct invocation of multi-container endpoints
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 74
Description
### Describe the feature
As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):
> [Direct Invocation of Multi-Container Endpoints][multi-container]: By default (and as [described in the proposed README](#inference-pipeline-model)), when a customer specifies multiple containers for a model, the containers are treated as an inference pipeline (also referred to as a serial pipeline). This means that the containers are treated as an ordered list, wherein the output of one container at runtime is passed as input to the next. Only the output from the last container is surfaced to the client invoking the model. To support a different invocation paradigm, the `InferenceExecutionConfig` structure was added to the model CloudFormation resource which allows customers to either explicitly configure `Serial` invocation mode (the default, as an inference pipeline) or the new `Direct` invocation mode. When using direct mode, a client invoking an endpoint must specify a container to target with their request; SageMaker then invokes only that single container.
[multi-container]: https://aws.amazon.com/blogs/machine-learning/deploy-multiple-serving-containers-on-a-single-instance-using-amazon-sagemaker-multi-container-endpoints/
[inference-pipeline-model]: https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#inference-pipeline-model
**Please 👍 this issue to help with the prioritization of this feature.**
### Use Case
"_This enables you to run up to 15 different ML containers on a single endpoint and invoke them independently, thereby saving up to 90% in costs. These ML containers can be running completely different ML frameworks and algorithms for model serving_." ([link](https://aws.amazon.com/blogs/machine-learning/deploy-multiple-serving-containers-on-a-single-instance-using-amazon-sagemaker-multi-container-endpoints/))
### Proposed Solution
As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):
> As SageMaker exposes a new dimension for CloudWatch metrics specific to each directly-invokable container, other than exposing a new inference execution mode attribute on the `Model` construct, this feature would likely also warrant the addition of a `findContainer(containerHostName: string)` method to [`IEndpointProductionVariant`](#endpoint-production-variants) which will return a new interface on which additional `metric*` APIs are present for generating CloudWatch metrics against the dimension consisting of endpoint, variant, and container combined.
[endpoint-production-variants]: https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#endpoint-production-variants
### Other Information
_No response_
### Acknowledgements
- [ ] I _may_ be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.54.0-alpha.0
### Environment details (OS name and version, etc.)
macOS Ventura
Contributor guide
Research direction
Start with the SageMaker Endpoint L2 construct RFC, especially its feature additions and endpoint-production-variants sections, then inspect the Model construct and IEndpointProductionVariant entry points. Done means supporting direct invocation mode and the proposed container-specific metric access without losing the existing serial behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100