aws / aws/sagemaker-huggingface-inference-toolkit

Support multiple return sequences

Open
#109 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
270
Forks
60
PR merge metrics
No merged PRs in 30d

Description

Is there any way to generate multiple return sequences for a text generation prompt? At the moment, I call the predictor sequentially n times.

I think the following linked issue would also solve my request as I could pass `num_return_sequences=n` as a kwarg to the HF pipeline.
https://github.com/aws/sagemaker-huggingface-inference-toolkit/issues/85

In the meantime, does `HuggingFacePredictor.predict` support batched inputs? This would be an improvement over my current implementation.

Contributor guide

Open the contributing guide

Research direction

Start at the HuggingFacePredictor.predict entry point and trace how prompts are passed to the Hugging Face pipeline. Determine whether batched inputs and multiple return sequences should be supported, then verify that one request can produce n results without sequential predictor calls; the linked issue 85 describes the related num_return_sequences request.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
backend, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.