Batch Inference Pipeline - Input/Output Sequence Handling and Prompt Failures guide
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
Hi, I have a question regarding **batch inference** in `lmdeploy` (using the Offline Inference Pipeline).
Suppose I set up a batch of, say, 32 prompts for inference:
```python
from lmdeploy import pipeline
pipe = pipeline('internlm/internlm2_5-7b-chat')
prompts = ['Hi, please introduce yourself', 'Shanghai is ...', ...] # batch of 32 prompts
response = pipe(prompts)
print(response)
```
My concern is about **tracking the output corresponding to each input prompt**:
1. Will the outputs be returned in the **same order** as the input prompts?
2. What happens if **one of the prompts fails** during inference—does it affect the order of outputs for the rest of the batch?
3. Is there a way to reliably **map each input prompt to its corresponding output**, even if some prompts fail?
Essentially, I want to ensure that for batch inference, I can always trace which output belongs to which input, without any sequence mismatch.
Could you clarify how `lmdeploy` handles this scenario?
Contributor guide
Assessment
This issue has not been assessed yet.