NVIDIA / NVIDIA/TensorRT-LLM

[feature request] /metrics endpoint disparity between trt and pytorch backend

Open
#6,364 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

I noticed the response returned by /metrics endpoint looks different between trt and pytorch backend (v1.0.0.rc4)

  • trt backend:
    {"cpuMemUsage":68172,"gpuMemUsage":36100332423,"inflightBatchingStats":{"avgNumDecodedTokensPerIter":1.0,"microBatchId":0,"numContextRequests":0,"numCtxTokens":0,"numGenRequests":1,"numPausedRequests":0,"numScheduledRequests":1},"iter":60,"iterLatencyMS":13.021209,"kvCacheStats":{"allocNewBlocks":3,"allocTotalBlocks":3,"cacheHitRate":0.0,"freeNumBlocks":22952,"maxNumBlocks":22955,"missedBlocks":2,"reusedBlocks":0,"tokensPerBlock":32,"usedNumBlocks":3},"maxBatchSizeRuntime":256,"maxBatchSizeStatic":2048,"maxBatchSizeTunerRecommended":256,"maxNumActiveRequests":2048,"maxNumTokensRuntime":8192,"maxNumTokensStatic":8192,"maxNumTokensTunerRecommended":0,"newActiveRequestsQueueLatencyMS":0.0,"numActiveRequests":1,"numCompletedRequests":0,"numNewActiveRequests":0,"numQueuedRequests":0,"pinnedMemUsage":2952841216,"specDecodingStats":null,"staticBatchingStats":null,"timestamp":"07-25-2025 16:54:34.140189"},

  • pytorch backend:
    {"cpuMemUsage":0,"gpuMemUsage":40560164864,"inflightBatchingStats":{"avgNumDecodedTokensPerIter":0.0,"microBatchId":0,"numContextRequests":0,"numCtxTokens":0,"numGenRequests":1,"numPausedRequests":0,"numScheduledRequests":1},"iter":587,"iterLatencyMS":3.3152103424072266,"kvCacheStats":{"allocNewBlocks":10740,"allocTotalBlocks":45247,"cacheHitRate":0.0,"freeNumBlocks":10082,"maxNumBlocks":10082,"missedBlocks":45231,"reusedBlocks":0,"tokensPerBlock":32,"usedNumBlocks":0},"maxBatchSizeRuntime":0,"maxBatchSizeStatic":0,"maxBatchSizeTunerRecommended":0,"maxNumActiveRequests":256,"maxNumTokensRuntime":0,"maxNumTokensStatic":0,"maxNumTokensTunerRecommended":0,"newActiveRequestsQueueLatencyMS":7.772445678710938e-05,"numActiveRequests":1,"numCompletedRequests":1,"numNewActiveRequests":0,"numQueuedRequests":0,"pinnedMemUsage":0,"specDecodingStats":null,"staticBatchingStats":{"emptyGenSlots":0,"numContextRequests":0,"numCtxTokens":0,"numGenTokens":0,"numScheduledRequests":0},"timestamp":"07-25-2025 17:07:28.471598","requestStats":[{"allocNewBlocksPerRequest":0,"allocTotalBlocksPerRequest":1,"avgNumDecodedTokensPerIter":0.998062014579773,"contextPrefillPosition":13,"disServingStats":null,"id":256,"kvCacheHitRatePerRequest":0.0,"missedBlocksPerRequest":1,"numGeneratedTokens":515,"paused":false,"reusedBlocksPerRequest":0,"scheduled":true,"stage":"GENERATION_COMPLETE"}]}

It seems that pytorch engine records (iteration_stats, request_stats) for each iteration as a tuple, while trt engine stores them separately and have diff methods to retrieve them.

I can see pros and cons for both approaches. As a user, I slightly lean to the way pytorch engine handles them - request_stats contains more detailed info. and I'd like to have it returned by default with the iteration_stats to better understand what's happening during a specific iteration.
Any plans to rework the iteration_stats logic for trt engine? (if no, I'd be more than happy to raise a PR :D)
or am I misunderstanding something here? Is there a recommended way to link iteration_stats and request_stats?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare the iteration and request statistics handling in tensorrt_llm/_torch/pyexecutor/py_executor.py around lines 714-717 and tensorrt_llm/executor/worker.py around lines 305-309. First determine how each backend currently associates these records, then decide whether the TRT response should expose request_stats with iteration_stats or provide another linkage. Done means the intended /metrics behavior is agreed and consistent for both backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
api, backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.