InternLM / InternLM/lmdeploy

Batch Inference Pipeline - Input/Output Sequence Handling and Prompt Failures guide

Open
#3,972 0 comments 0 reactions 1 assignee Claimed by @lvhan028 View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

Hi, I have a question regarding **batch inference** in `lmdeploy` (using the Offline Inference Pipeline).

Suppose I set up a batch of, say, 32 prompts for inference:

```python
from lmdeploy import pipeline

pipe = pipeline('internlm/internlm2_5-7b-chat')
prompts = ['Hi, please introduce yourself', 'Shanghai is ...', ...] # batch of 32 prompts
response = pipe(prompts)
print(response)
```

My concern is about **tracking the output corresponding to each input prompt**:

1. Will the outputs be returned in the **same order** as the input prompts?
2. What happens if **one of the prompts fails** during inference—does it affect the order of outputs for the rest of the batch?
3. Is there a way to reliably **map each input prompt to its corresponding output**, even if some prompts fail?

Essentially, I want to ensure that for batch inference, I can always trace which output belongs to which input, without any sequence mismatch.

Could you clarify how `lmdeploy` handles this scenario?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.