aws / aws/sagemaker-huggingface-inference-toolkit
Support multiple return sequences
- Dominant language
- Python
- Stars
- 270
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
Is there any way to generate multiple return sequences for a text generation prompt? At the moment, I call the predictor sequentially n times.
I think the following linked issue would also solve my request as I could pass `num_return_sequences=n` as a kwarg to the HF pipeline.
https://github.com/aws/sagemaker-huggingface-inference-toolkit/issues/85
In the meantime, does `HuggingFacePredictor.predict` support batched inputs? This would be an improvement over my current implementation.
Contributor guide
Research direction
Start at the HuggingFacePredictor.predict entry point and trace how prompts are passed to the Hugging Face pipeline. Determine whether batched inputs and multiple return sequences should be supported, then verify that one request can produce n results without sequential predictor calls; the linked issue 85 describes the related num_return_sequences request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- backend, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100