aws / aws/sagemaker-huggingface-inference-toolkit

How to enable Batch inference on AWS deployed Serverless model from Hub?

Open
#98 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
270
Forks
60
PR merge metrics
No merged PRs in 30d

Description

I am using the serverless inference from Sagemaker with Huggingface Model from the hub
according this example :
https://github.com/huggingface/notebooks/blob/main/sagemaker/19_serverless_inference/sagemaker-notebook.ipynb

using the
#image uri
image_container=get_huggingface_llm_image_uri("huggingface",version="0.9.3")

I was expecting the resulting pipeline to execute as the Pipeline class from transformers for this task (text generation)
however, the input does not work with list.

Is there any approach to do batch inference on Sagemaker SDK?

Contributor guide

Open the contributing guide

Research direction

Start with the linked sagemaker/19_serverless_inference/sagemaker-notebook.ipynb and the SageMaker SDK path used for the serverless Hugging Face model. Check how list inputs are handled for text generation and document whether batch inference is supported; done means a verified batch-inference approach or a clearly recorded limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, huggingface, python
Domain
cloud, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.