aws / aws/aws-cdk

sagemaker: Support direct invocation of multi-container endpoints

Open
#23,155 1 comment 1 reaction 0 assignees View on GitHub
@aws-cdk/aws-sagemaker effort/small feature-request p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
74

Description

### Describe the feature

As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):

> [Direct Invocation of Multi-Container Endpoints][multi-container]: By default (and as [described in the proposed README](#inference-pipeline-model)), when a customer specifies multiple containers for a model, the containers are treated as an inference pipeline (also referred to as a serial pipeline). This means that the containers are treated as an ordered list, wherein the output of one container at runtime is passed as input to the next. Only the output from the last container is surfaced to the client invoking the model. To support a different invocation paradigm, the `InferenceExecutionConfig` structure was added to the model CloudFormation resource which allows customers to either explicitly configure `Serial` invocation mode (the default, as an inference pipeline) or the new `Direct` invocation mode. When using direct mode, a client invoking an endpoint must specify a container to target with their request; SageMaker then invokes only that single container.

[multi-container]: https://aws.amazon.com/blogs/machine-learning/deploy-multiple-serving-containers-on-a-single-instance-using-amazon-sagemaker-multi-container-endpoints/
[inference-pipeline-model]: https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#inference-pipeline-model

**Please 👍 this issue to help with the prioritization of this feature.**

### Use Case

"_This enables you to run up to 15 different ML containers on a single endpoint and invoke them independently, thereby saving up to 90% in costs. These ML containers can be running completely different ML frameworks and algorithms for model serving_." ([link](https://aws.amazon.com/blogs/machine-learning/deploy-multiple-serving-containers-on-a-single-instance-using-amazon-sagemaker-multi-container-endpoints/))

### Proposed Solution

As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):

> As SageMaker exposes a new dimension for CloudWatch metrics specific to each directly-invokable container, other than exposing a new inference execution mode attribute on the `Model` construct, this feature would likely also warrant the addition of a `findContainer(containerHostName: string)` method to [`IEndpointProductionVariant`](#endpoint-production-variants) which will return a new interface on which additional `metric*` APIs are present for generating CloudWatch metrics against the dimension consisting of endpoint, variant, and container combined.

[endpoint-production-variants]: https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#endpoint-production-variants

### Other Information

_No response_

### Acknowledgements

- [ ] I _may_ be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.54.0-alpha.0

### Environment details (OS name and version, etc.)

macOS Ventura

Contributor guide

Open the contributing guide

Research direction

Start with the SageMaker Endpoint L2 construct RFC, especially its feature additions and endpoint-production-variants sections, then inspect the Model construct and IEndpointProductionVariant entry points. Done means supporting direct invocation mode and the proposed container-specific metric access without losing the existing serial behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.