aws / aws/sagemaker-huggingface-inference-toolkit
How to enable Batch inference on AWS deployed Serverless model from Hub?
- Dominant language
- Python
- Stars
- 270
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
I am using the serverless inference from Sagemaker with Huggingface Model from the hub
according this example :
https://github.com/huggingface/notebooks/blob/main/sagemaker/19_serverless_inference/sagemaker-notebook.ipynb
using the
#image uri
image_container=get_huggingface_llm_image_uri("huggingface",version="0.9.3")
I was expecting the resulting pipeline to execute as the Pipeline class from transformers for this task (text generation)
however, the input does not work with list.
Is there any approach to do batch inference on Sagemaker SDK?
Contributor guide
Research direction
Start with the linked sagemaker/19_serverless_inference/sagemaker-notebook.ipynb and the SageMaker SDK path used for the serverless Hugging Face model. Check how list inputs are handled for text generation and document whether batch inference is supported; done means a verified batch-inference approach or a clearly recorded limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, huggingface, python
- Domain
- cloud, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100