aws / aws/sagemaker-python-sdk

Create Awaitable predict capability

Open
#3,973 10 comments 5 reactions 0 assignees View on GitHub
component: async inference type: feature request
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the feature you'd like**
Like many other inference libraries in python (e.g. OpenAI), create a real awaitable version of Predict for realtime sagemaker inference endpoints. This will help python applications that use FastAPI and asyncio to deliver realtime responses while not blocking the main event loop. Please note that this feature is different that the one currently available [here](https://sagemaker.readthedocs.io/en/v2.169.0/api/inference/predictor_async.html) where the predictions are written to a S3 bucket. This feature would work exactly like https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html#sagemaker.predictor.Predictor.predict but with an `await` in real `asyncio` style.

Sagemaker is an amazing library and it would be just way better for production environments using FastAPI to have this feature.

**How would this feature be used? Please describe.**
In this case, currently, the sync version looks like this:
```python
response = predictor.predict(input_data)
```
The async might be looking like
```python
response = await predictor.apredict(input_data)
```

**Describe alternatives you've considered**
I considered subclassing the `predictor` and add the async version.

**Additional context**
For modern python applications building on top of FastAPI and Asyncio, it is crucial to use async modalities do avoid blocking the main event-loop in the server (in case of scalable applications). Therefore, having a real `awaitable` functionality would avoid blocking the main event loop of the applications that leverage sagemaker.

Thanks alot

Contributor guide

Open the contributing guide

Research direction

Start with the documented Predictor.predict behavior and compare it with the existing predictor_async capability linked in the issue, which writes predictions to S3. Trace how realtime SageMaker endpoint calls are made and determine the interface needed for an awaitable equivalent. Done means Python asyncio applications can await realtime predictions without using S3-backed async behavior or blocking the event loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.