sagemaker: Support asynchronous endpoints
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the feature
As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):
> [Asynchronous Inference][async-inference]: By default, a deployed endpoint is synchronous: a customer issues an InvokeEndpoint operation to SageMaker with an attached input payload and the resulting response contains the output payload from the endpoint. To instead support asynchronous invocation, the `AsyncInferenceClientConfig` CloudFormation attribute was added to the endpoint config resource. To interact with an asynchronous endpoint, a customer issues an InvokeEndpointAsync operation to SageMaker with an attached input location in S3; SageMaker asynchronously reads the input from S3, invokes the endpoint, and writes the output to an S3 location specified within the `AsyncInferenceClientConfig` attribute.
[async-inference]: https://aws.amazon.com/about-aws/whats-new/2021/08/amazon-sagemaker-asynchronous-new-inference-option/
**Please 👍 this issue to help with the prioritization of this feature.**
### Use Case
"_This option is ideal for requests with large payload sizes (up to 1GB), long processing times (up to 15 minutes), and near real-time latency requirements. Asynchronous Inference enables you to save on costs by autoscaling the instance count to zero when there are no requests to process, so you only pay when your endpoint is processing requests_." ([link](https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html))
### Proposed Solution
As described in [the SageMaker `Endpoint` L2 construct RFC](https://github.com/aws/aws-cdk-rfcs/blob/master/text/0431-sagemaker-l2-endpoint.md#feature-additions):
> As [discussed with the RFC bar raiser here][design-conversation], there are a few ways to tackle the addition of this functionlity. One option is to add attribute(s) to the L2 endpoint config construct to support asynchronous inference along with synthesis-time error handling to catch configuration conflicts (e.g., asynchronous endpoints are only capable of supporting a single instance-based production variant today). Alternatively, an `AsyncEndpointConfig` subclass of `EndpointConfig` could be introduced to provide a better compile-time contract to customers (while still implementing the generic functionality within `EndpointConfig`). Either way, the proposed contracts would only undergo backward-compatible changes.
[design-conversation]: https://github.com/aws/aws-cdk-rfcs/pull/433#discussion_r952949608
### Other Information
_No response_
### Acknowledgements
- [ ] I _may_ be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.54.0-alpha.0
### Environment details (OS name and version, etc.)
macOS Ventura
Contributor guide
Research direction
Start with the SageMaker Endpoint L2 construct RFC, then inspect the existing Endpoint and EndpointConfig constructs and the CloudFormation AsyncInferenceClientConfig attribute. Done means asynchronous invocation can be configured with the documented constraints and the resulting endpoint configuration synthesizes correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100