Add Parameter Validation to trtllm-serve to Prevent Server Crashes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Description
Currently, the trtllm-serve lacks proper validation for client input parameters. When invalid values are provided (e.g., top_k = -1 or input sequences exceeding max_seq_len), the server crashes instead of returning an error response. This behavior severely impacts availability, as a single malformed request can disrupt service for all clients.
Reproduction Steps
- Send a request with top_k = -1 (default value in vLLM)
- Send a request with an input sequence longer than
max_seq_len.
Expected Behavior
- The server should return a 400 Bad Request error with details (e.g., "top_k must be >= 0" or "Input length exceeds max_seq_len").
- The server must not crash and should continue processing other requests.
Actual Behavior
The server crashes with unhandled exceptions (e.g., RuntimeError or assertion failures), requiring a restart.
Proposed Solution
Implement validation checks:
- Validation in OpenAI Protocol (
tensorrt_llm/serve/openai_protocol.py)
Add parameter checks(top_k, top_p, max_tokens, etc.) toCompletionRequestandChatCompletionRequestdataclasses:
top_k: int = Field(default=0, ge=-1)
top_p: Optional[float] = Field(default=1.0, gt=0.0, le=1.0)
...
- Input Length Validation in LLM(
tensorrt_llm/llmapi/llm.py)
InBaseLLM.generate_async, add explicit input length checks before passing request to the executor:
def generate_async(
self,
inputs: PromptInputs,
sampling_params: Optional[SamplingParams] = None,
lora_request: Optional[LoRARequest] = None,
prompt_adapter_request: Optional[PromptAdapterRequest] = None,
streaming: bool = False,
kv_cache_retention_config: Optional[KvCacheRetentionConfig] = None,
disaggregated_params: Optional[DisaggregatedParams] = None,
_postproc_params: Optional[PostprocParams] = None,
) -> RequestOutput:
...
if self.args.max_seq_len <= len(prompt_token_ids):
raise Exception(f"Input length exceeds max sequence length, {len(prompt_token_ids)} >= {self.args.max_seq_len}")
...
Additional Context
- This issue is critical for production deployments where malformed requests are common (e.g., misconfigured clients).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tensorrt_llm/serve/openai_protocol.py, reviewing CompletionRequest and ChatCompletionRequest validation, then inspect tensorrt_llm/llmapi/llm.py at BaseLLM.generate_async. Reproduce invalid top_k values and inputs exceeding max_seq_len. Done means invalid requests return detailed 400 errors without crashing the server or disrupting subsequent requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100