NVIDIA / NVIDIA/TensorRT-LLM

Add Parameter Validation to trtllm-serve to Prevent Server Crashes

Open
#6,329 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

Description
Currently, the trtllm-serve lacks proper validation for client input parameters. When invalid values are provided (e.g., top_k = -1 or input sequences exceeding max_seq_len), the server crashes instead of returning an error response. This behavior severely impacts availability, as a single malformed request can disrupt service for all clients.

Reproduction Steps

  • Send a request with top_k = -1 (default value in vLLM)
  • Send a request with an input sequence longer than max_seq_len.

Expected Behavior

  • The server should return a 400 Bad Request error with details (e.g., "top_k must be >= 0" or "Input length exceeds max_seq_len").
  • The server must not crash and should continue processing other requests.

Actual Behavior
The server crashes with unhandled exceptions (e.g., RuntimeError or assertion failures), requiring a restart.

Proposed Solution
Implement validation checks:

  1. Validation in OpenAI Protocol (tensorrt_llm/serve/openai_protocol.py)
    Add parameter checks(top_k, top_p, max_tokens, etc.) to CompletionRequest and ChatCompletionRequest dataclasses:
top_k: int = Field(default=0, ge=-1) 
top_p: Optional[float] = Field(default=1.0, gt=0.0, le=1.0)
...
  1. Input Length Validation in LLM(tensorrt_llm/llmapi/llm.py)
    In BaseLLM.generate_async, add explicit input length checks before passing request to the executor:
def generate_async(
        self,
        inputs: PromptInputs,
        sampling_params: Optional[SamplingParams] = None,
        lora_request: Optional[LoRARequest] = None,
        prompt_adapter_request: Optional[PromptAdapterRequest] = None,
        streaming: bool = False,
        kv_cache_retention_config: Optional[KvCacheRetentionConfig] = None,
        disaggregated_params: Optional[DisaggregatedParams] = None,
        _postproc_params: Optional[PostprocParams] = None,
    ) -> RequestOutput:
     ...
     if self.args.max_seq_len <= len(prompt_token_ids):
          raise Exception(f"Input length exceeds max sequence length, {len(prompt_token_ids)} >= {self.args.max_seq_len}")
     ...

Additional Context

  • This issue is critical for production deployments where malformed requests are common (e.g., misconfigured clients).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tensorrt_llm/serve/openai_protocol.py, reviewing CompletionRequest and ChatCompletionRequest validation, then inspect tensorrt_llm/llmapi/llm.py at BaseLLM.generate_async. Reproduce invalid top_k values and inputs exceeding max_seq_len. Done means invalid requests return detailed 400 errors without crashing the server or disrupting subsequent requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.