[RFC]: Disaggregated request preprocessing

Motivation

Dynamo's standalone EPP DEP, llm-d, and other disaggregated serving frameworks place routing outside the inference engine. Disaggregated routers, including endpoint pickers (EPPs) adhering to Gateway API Inference Extension (GAIE), need the exact prompt token IDs the selected engine will execute for cache-aware routing. They use these IDs to measure request length and compare prompt prefixes with cached blocks reported through KV events.

Differences in chat templates, special tokens, or model-specific formatting can produce different token sequences and incorrect cache-overlap scores. TensorRT-LLM's existing /_internal/tokenize endpoint accepts plain text, not a full chat request, and runs within the normal inference server. Callers still need to reproduce TensorRT-LLM's request preparation themselves.

Exposing the serving preprocessing path as a CPU-only service would let routers obtain accurate model inputs and let deployments prepare requests once, independently of GPU execution. Other engines provide useful design and implementation precedents:

• vLLM: the disaggregated serving RFC and Renderer RFC, with the merged CPU-only render service.
• SGLang: the disaggregated request preprocessing RFC and its standalone renderer implementation PR.

TensorRT-LLM should expose the same separation while retaining its own preprocessing semantics and generate-request format.

Proposed Change

Introduce a TensorRT-LLM renderer that converts chat and completion requests into a prepared generate request without running inference. Normal serving and standalone rendering must call the same preprocessing implementation, including request validation, chat templates, tools, reasoning options, and model-specific token handling.

For the same request and model/tokenizer configuration, the renderer must return exactly the token sequence passed to generation. The standalone service loads only the configuration, tokenizer, and preprocessing assets it needs; it must start on a CPU-only host without model weights or an inference engine.

Current state

Request preparation is split between the serving layer and the engine-owning LLM object:

flowchart TD
    R["Chat-completions request"] --> A

    subgraph Server["OpenAIServer"]
        A["Request preparation<br/>Validation, templates, tools<br/>and model-specific formatting"]
    end

    subgraph LLM["LLM — preprocessing and execution share one lifecycle"]
        B["preprocess() / _preprocess()<br/>Tokenization and model input processing"]
        C["PreprocessedInputs<br/>Token IDs + optional multimodal inputs"]
        D["generate_async()<br/>Resolve parameters, validate and submit"]
        E["GenerationExecutor<br/>Scheduling, KV cache and GPU execution"]
        B --> C --> D --> E
    end

    A -->|"Prepared prompt"| B
    A -->|"Sampling and other request options"| D

• The server obtains preprocessing resources through its generator. LLM initialization loads the tokenizer and input processor and creates the executor.
• PreprocessedInputs already lets generation skip preprocessing, but generation settings and request metadata are passed separately. It is not yet a complete portable generate request.

Proposed refactoring

Extract both preparation stages into a shared CPU renderer, used by normal serving and standalone rendering:

flowchart TD
    N["Normal chat/completion serving"] --> A
    R["Standalone render HTTP / Python API"] --> A

    subgraph Renderer["Shared CPU renderer — owns tokenizer, config and templates"]
        A["API request preparation<br/>Validation, templates, tools and model semantics"]
        B["Model input preprocessing<br/>Canonical token IDs and prepared inputs"]
        A --> B
    end

    B --> Q["GenerateRequest<br/>Token IDs + generation settings + metadata<br/>Multimodal inputs later"]
    Q -->|"Local call"| X
    Q -->|"Render response"| C["Caller / router"]
    C -->|"POST /generate"| X

    subgraph Worker["Engine worker"]
        X["Execution entrypoint<br/>Validate and translate prepared request"]
        G["Existing GenerationExecutor<br/>Scheduling, KV cache and GPU execution"]
        X --> G
    end

• Move API request preparation out of OpenAIServer and input preprocessing out of the LLM lifecycle. Load their CPU resources independently of the executor.
• Combine prepared inputs and generation settings into GenerateRequest, usable through either a local call or HTTP.
• Keep execution validation and submission on the worker, reusing the existing prepared-input path without repeating templates or tokenization.

Python and CLI

Expose the renderer as a reusable Python API independent of an initialized LLM. Provide a standalone server through trtllm-render --model MODEL, also invocable as python -m tensorrt_llm.serve.render --model MODEL. These names are proposed.

HTTP APIs

Both routes use the same request semantics as their inference counterparts and are available in standalone mode and normal serving.

Delivery

1. Text first: share the existing preprocessing path and expose canonical prompt token IDs through the Python API and HTTP service. This immediately supports routing for text-only requests.
2. Direct execution: complete the generate-request contract with generation settings and required request metadata. A direct generation endpoint, provisionally POST /generate, consumes this output without repeating chat formatting or tokenization.
3. Multimodal extension: add the multimodal features and metadata needed by generation, including media identity and positions in the token sequence. Keep CPU preprocessing separate from GPU encoder execution.

Validate token equality against normal serving and startup on a host without GPUs. Initially reject unsupported multimodal requests rather than returning approximate inputs.

Feedback Period

At least one week from posting.

CC List

• @NVIDIA/trt-llm-runtime-devs — serving, input processing, and LLM API ownership.
• @QiJune @WeiHaocheng @KleinBlueC — reviewers and author of the existing tokenization endpoint.