vllm-project / vllm-project/aibrix

Missing finish_reason check in pd_disaggregation for trtllm

Open
#2,013 2 comments 0 reactions 1 assignee Claimed by @scarlet25151 View on GitHub
area/gateway kind/enhancement priority/important-soon
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

TRT-LLM's proxy does `_verify_ctx_response()`(https://github.com/NVIDIA/TensorRT-LLM/blob/ae76df1615e0710c6b97a62b2e31c36c3e0f550c/tensorrt_llm/serve/openai_disagg_service.py#L339) which checks: if finish_reason is NOT "length" or "not_finished", skip the generation phase entirely (context already completed within max_tokens=1). The current code always proceeds to decode regardless.

### Use Case

We do have some tasks only generate 1 token, for example, yes or no, numbers etc. In this case, it may throw errors. Let's make sure the workflow cover such cases.

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.