[RFC]: ForwardPassMetrics — per-iteration scheduler telemetry for dynamo integration

Motivation

Dynamo needs per-iteration scheduler telemetry from all backends (vLLM, TRT-LLM, SGLang) for performance modeling and load-aware routing. This telemetry is called ForwardPassMetrics (FPM) — it reports what the engine did each forward pass: prefill/decode request counts, token counts, KV cache pressure, queue depth, and iteration wall time.

FPM is already implemented for vLLM in the dynamo repo:

• ai-dynamo/dynamo#7200 — vLLM scheduler emits FPM via ZMQ PUB
• ai-dynamo/dynamo#7250 — dynamo parent process relays FPM to event plane (NATS)
• ai-dynamo/dynamo#7537 — fix for async scheduling

The forward_pass_metrics.py in dynamo explicitly notes: TODO: add metrics for TrtLLM/SGLang.

This RFC proposes adding FPM emission to TRT-LLM's PyTorch backend (PyExecutor).

FPM Metric Schema

Per-iteration message (one per forward pass):

ForwardPassMetrics:
  worker_id:                           # dynamo worker id, matching info in MDC
  dp_rank:                             # each dp rank has its own scheduler
  wall_time:                           # wall time of this iteration

  scheduled_requests:                   # requests scheduled in this iteration
    num_prefill_requests:
    sum_prefill_tokens:                 # tokens freshly computed (not cached)
    var_prefill_length:                 # variance of full prompt lengths
    sum_prefill_tokens_prefix_cached:   # † KV tokens from prefix cache
    var_non_prefix_cached_prefill_length: # † variance of non-cached prefill lengths
    num_decode_requests:
    sum_decode_kv_tokens:               # total KV context length across decode requests
    sum_decode_kv_tokens_prefix_cached: # † cached KV tokens in decode
    var_decode_kv_tokens:
    var_decode_kv_tokens_prefix_cached: # † variance of cached decode KV lengths

  queued_requests:                      # requests exist but not scheduled this iteration
    num_prefill_requests:
    sum_prefill_tokens:
    var_prefill_length:
    num_decode_requests:
    sum_decode_kv_tokens:
    var_decode_kv_tokens:

† = fields from the broader dynamo proposal, not yet in the current dynamo implementation but planned.

The current dynamo implementation (dynamo.common.forward_pass_metrics) uses msgspec.Struct with msgpack serialization. It includes version, counter_id fields plus the non-† fields (with slightly different naming: sum_prefill_kv_tokens instead of sum_prefill_tokens_prefix_cached).

TRT-LLM Architecture Analysis

Key findings from codebase exploration:

1. PyExecutor runs in a subprocess — GenerationExecutorProxy spawns workers via MpiPoolSession (mpi4py). This is the same architecture as vLLM's forked EngineCore child process. The dynamo runtime (NATS, tokio) is NOT available in the subprocess.

2. ZMQ is already a TRT-LLM dependency — used extensively for RPC (ipc.py, rpc_server.py, worker communication).

3. IterationStats is a C++ struct — defined in types.h, bound via nanobind, serialized via NLOHMANN JSON. Extending it requires C++ changes + rebuild.

4. ScheduledRequests has clean prefill/decode separation — context_requests_chunking, context_requests_last_chunk, generation_requests, paused_requests. All FPM fields are computable from LlmRequest properties (orig_prompt_len, context_current_position, max_beam_num_tokens, estimated_reusable_tokens, etc.).

5. waiting_queue is iterable from the executor thread — can classify queued requests as prefill vs decode.

6. Existing iter_stats captures ~70% of FPM fields — num_context_requests, num_gen_requests, num_ctx_tokens, iter_latency_ms, num_queued_requests. But iter_stats goes through C++ serialization and can't be extended without C++ changes.

7. Prefix cache info is available — estimated_reusable_tokens on LlmRequest tracks prefix cache hits. context_current_position tracks previously-computed chunks.

Proposed Approaches

Option A: New ZMQ PUB channel (matching vLLM pattern)

Worker subprocess (PyExecutor)          Parent process (dynamo)
  _executor_loop()
    ├── schedule → forward → sample
    ├── compute FPM from ScheduledRequests
    │     └── queue.put(metrics) (~1μs)
    └── _FpmPublisherThread
          └── ZMQ PUB ──────────────→ FpmEventRelay (ZMQ SUB → NATS)
              tcp://*:{port}            (existing Rust binding, backend-agnostic)

• Activation: DYN_FORWARDPASS_METRIC_PORT env var (same as vLLM)
• Serialization: msgspec msgpack (conditional import from dynamo.common.forward_pass_metrics)
• New file: tensorrt_llm/_torch/pyexecutor/forward_pass_metrics.py
• Modified: py_executor.py (~10 lines)

Pro	Con
Dynamo's FpmEventRelay works identically for vLLM and TRT-LLM	New ZMQ socket + port per executor
Push-based (immediate delivery)	Adds DYN_FORWARDPASS_METRIC_PORT env var
Matches vLLM pattern exactly — same consumer code	Conditional import from dynamo packages
Zero overhead when disabled (env var not set)

Option B: Reuse existing RPC stats infrastructure

Worker subprocess (PyExecutor)          Parent process
  _executor_loop()
    ├── compute FPM alongside _update_iter_stats()
    ├── store in self._fpm_buffer (Python deque)
    │
    │     ←── RPC fetch_fpm() ────── dynamo Publisher polls
    └── return FPM data via RPC ───→ dynamo converts to ForwardPassMetrics

• Activation: new RPC method + config flag
• Serialization: JSON (matching existing stats path) or custom
• Modified: py_executor.py, rpc_server.py, base_worker.py

Pro	Con
No new sockets or ports	Polled (10-100ms backoff latency)
Reuses existing ZMQ PAIR RPC channel	Dynamo needs TRT-LLM-specific FPM consumer (can't reuse FpmEventRelay)
Familiar pattern (like get_stats/get_kv_cache_events)	Different wire format than vLLM FPM
No conditional dynamo import	More files modified (RPC server, base_worker, proxy)

Option C: Hybrid — compute alongside iter_stats, emit via ZMQ PUB

• Same ZMQ PUB emission as Option A
• But FPM computation co-located with existing _update_iter_stats()
• Single point of metric extraction, reuses overlapping field computations

Pro	Con
Single computation point (no duplicate iteration)	Still needs new ZMQ socket
Push-based + vLLM-compatible wire format	Mixes concerns in _update_iter_stats
Reuses existing stats computation for overlapping fields

Data Availability Mapping

FPM Field	TRT-LLM Source	In iter_stats?
num_prefill_requests	scheduled_batch.num_context_requests	✅
sum_prefill_tokens	model_engine.iter_states['num_ctx_tokens']	✅
num_decode_requests	scheduled_batch.num_generation_requests	✅
wall_time	iter_latency_ms / 1000	✅
sum_decode_kv_tokens	sum(req.max_beam_num_tokens for gen_requests)	❌
var_prefill_length	Welford over req.orig_prompt_len	❌
var_decode_kv_tokens	Welford over req.max_beam_num_tokens	❌
sum_prefill_tokens_prefix_cached †	sum(req.estimated_reusable_tokens for ctx_requests)	❌
Queued prefill vs decode split	Iterate waiting_queue, classify by state	❌ (only total count)

† = planned, not yet in dynamo's current schema

Open Questions

1. Transport mechanism: Should FPM use a new ZMQ PUB channel (Option A, matching vLLM) or the existing RPC stats path (Option B)? Option A enables backend-agnostic consumer code in dynamo. Option B reuses existing IPC infrastructure.

2. Schema version: Should the initial TRT-LLM implementation match the current dynamo schema (without prefix-cached fields), or include the full proposal schema from the start?

3. msgspec dependency: TRT-LLM doesn't have msgspec. Options:

   • Conditional from dynamo.common.forward_pass_metrics import ... (works when dynamo launches TRT-LLM)
   • Add msgspec to requirements.txt
   • Use msgpack directly to produce compatible bytes

4. Single-process mode: When enable_worker_single_process_for_tp1() is set, PyExecutor is NOT in a subprocess. Should FPM still use ZMQ PUB (works fine same-process) or switch to a lighter mechanism?

CC List

@tedzhouhk (ForwardPassMetrics author)

References

• vLLM InstrumentedScheduler: components/src/dynamo/vllm/instrumented_scheduler.py in dynamo repo
• ForwardPassMetrics schema: components/src/dynamo/common/forward_pass_metrics.py in dynamo repo
• TRT-LLM PyExecutor: tensorrt_llm/_torch/pyexecutor/py_executor.py
• TRT-LLM IterationStats: cpp/include/tensorrt_llm/executor/types.h

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start with tensorrt_llm/_torch/pyexecutor/py_executor.py and compare its iteration and RPC paths with cpp/include/tensorrt_llm/executor/types.h and the referenced Dynamo ForwardPassMetrics files. Resolve the transport, schema, dependency, and single-process questions, then validate that per-iteration metrics can be emitted for the required scheduler fields without disrupting existing execution.