aws / aws/aws-cdk

sagemaker: Support asynchronous endpoints

Open
#23,149 1 comment 16 reactions 0 assignees View on GitHub
@aws-cdk/aws-sagemaker effort/medium feature-request p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the feature

As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):

> [Asynchronous Inference][async-inference]: By default, a deployed endpoint is synchronous: a customer issues an InvokeEndpoint operation to SageMaker with an attached input payload and the resulting response contains the output payload from the endpoint. To instead support asynchronous invocation, the `AsyncInferenceClientConfig` CloudFormation attribute was added to the endpoint config resource. To interact with an asynchronous endpoint, a customer issues an InvokeEndpointAsync operation to SageMaker with an attached input location in S3; SageMaker asynchronously reads the input from S3, invokes the endpoint, and writes the output to an S3 location specified within the `AsyncInferenceClientConfig` attribute.

[async-inference]: https://aws.amazon.com/about-aws/whats-new/2021/08/amazon-sagemaker-asynchronous-new-inference-option/

**Please 👍 this issue to help with the prioritization of this feature.**

### Use Case

"_This option is ideal for requests with large payload sizes (up to 1GB), long processing times (up to 15 minutes), and near real-time latency requirements. Asynchronous Inference enables you to save on costs by autoscaling the instance count to zero when there are no requests to process, so you only pay when your endpoint is processing requests_." ([link](https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html))

### Proposed Solution

As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):

> As [discussed with the RFC bar raiser here][design-conversation], there are a few ways to tackle the addition of this functionlity. One option is to add attribute(s) to the L2 endpoint config construct to support asynchronous inference along with synthesis-time error handling to catch configuration conflicts (e.g., asynchronous endpoints are only capable of supporting a single instance-based production variant today). Alternatively, an `AsyncEndpointConfig` subclass of `EndpointConfig` could be introduced to provide a better compile-time contract to customers (while still implementing the generic functionality within `EndpointConfig`). Either way, the proposed contracts would only undergo backward-compatible changes.

[design-conversation]: https://github.com/aws/aws-cdk-rfcs/pull/433#discussion_r952949608

### Other Information

_No response_

### Acknowledgements

- [ ] I _may_ be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.54.0-alpha.0

### Environment details (OS name and version, etc.)

macOS Ventura

Contributor guide

Open the contributing guide

Research direction

Start with the SageMaker Endpoint L2 construct RFC, then inspect the existing Endpoint and EndpointConfig constructs and the CloudFormation AsyncInferenceClientConfig attribute. Done means asynchronous invocation can be configured with the documented constraints and the resulting endpoint configuration synthesizes correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.