vllm-project / vllm-project/aibrix
Missing finish_reason check in pd_disaggregation for trtllm
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
TRT-LLM's proxy does `_verify_ctx_response()`(https://github.com/NVIDIA/TensorRT-LLM/blob/ae76df1615e0710c6b97a62b2e31c36c3e0f550c/tensorrt_llm/serve/openai_disagg_service.py#L339) which checks: if finish_reason is NOT "length" or "not_finished", skip the generation phase entirely (context already completed within max_tokens=1). The current code always proceeds to decode regardless.
### Use Case
We do have some tasks only generate 1 token, for example, yes or no, numbers etc. In this case, it may throw errors. Let's make sure the workflow cover such cases.
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.