[feature request] /metrics endpoint disparity between trt and pytorch backend
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
I noticed the response returned by /metrics endpoint looks different between trt and pytorch backend (v1.0.0.rc4)
-
trt backend:
{"cpuMemUsage":68172,"gpuMemUsage":36100332423,"inflightBatchingStats":{"avgNumDecodedTokensPerIter":1.0,"microBatchId":0,"numContextRequests":0,"numCtxTokens":0,"numGenRequests":1,"numPausedRequests":0,"numScheduledRequests":1},"iter":60,"iterLatencyMS":13.021209,"kvCacheStats":{"allocNewBlocks":3,"allocTotalBlocks":3,"cacheHitRate":0.0,"freeNumBlocks":22952,"maxNumBlocks":22955,"missedBlocks":2,"reusedBlocks":0,"tokensPerBlock":32,"usedNumBlocks":3},"maxBatchSizeRuntime":256,"maxBatchSizeStatic":2048,"maxBatchSizeTunerRecommended":256,"maxNumActiveRequests":2048,"maxNumTokensRuntime":8192,"maxNumTokensStatic":8192,"maxNumTokensTunerRecommended":0,"newActiveRequestsQueueLatencyMS":0.0,"numActiveRequests":1,"numCompletedRequests":0,"numNewActiveRequests":0,"numQueuedRequests":0,"pinnedMemUsage":2952841216,"specDecodingStats":null,"staticBatchingStats":null,"timestamp":"07-25-2025 16:54:34.140189"}, -
pytorch backend:
{"cpuMemUsage":0,"gpuMemUsage":40560164864,"inflightBatchingStats":{"avgNumDecodedTokensPerIter":0.0,"microBatchId":0,"numContextRequests":0,"numCtxTokens":0,"numGenRequests":1,"numPausedRequests":0,"numScheduledRequests":1},"iter":587,"iterLatencyMS":3.3152103424072266,"kvCacheStats":{"allocNewBlocks":10740,"allocTotalBlocks":45247,"cacheHitRate":0.0,"freeNumBlocks":10082,"maxNumBlocks":10082,"missedBlocks":45231,"reusedBlocks":0,"tokensPerBlock":32,"usedNumBlocks":0},"maxBatchSizeRuntime":0,"maxBatchSizeStatic":0,"maxBatchSizeTunerRecommended":0,"maxNumActiveRequests":256,"maxNumTokensRuntime":0,"maxNumTokensStatic":0,"maxNumTokensTunerRecommended":0,"newActiveRequestsQueueLatencyMS":7.772445678710938e-05,"numActiveRequests":1,"numCompletedRequests":1,"numNewActiveRequests":0,"numQueuedRequests":0,"pinnedMemUsage":0,"specDecodingStats":null,"staticBatchingStats":{"emptyGenSlots":0,"numContextRequests":0,"numCtxTokens":0,"numGenTokens":0,"numScheduledRequests":0},"timestamp":"07-25-2025 17:07:28.471598","requestStats":[{"allocNewBlocksPerRequest":0,"allocTotalBlocksPerRequest":1,"avgNumDecodedTokensPerIter":0.998062014579773,"contextPrefillPosition":13,"disServingStats":null,"id":256,"kvCacheHitRatePerRequest":0.0,"missedBlocksPerRequest":1,"numGeneratedTokens":515,"paused":false,"reusedBlocksPerRequest":0,"scheduled":true,"stage":"GENERATION_COMPLETE"}]}
It seems that pytorch engine records (iteration_stats, request_stats) for each iteration as a tuple, while trt engine stores them separately and have diff methods to retrieve them.
- https://github.com/NVIDIA/TensorRT-LLM/blob/v1.0.0rc4/tensorrt_llm/_torch/pyexecutor/py_executor.py#L714-L717
- https://github.com/NVIDIA/TensorRT-LLM/blob/v1.0.0rc4/tensorrt_llm/executor/worker.py#L305-L309
I can see pros and cons for both approaches. As a user, I slightly lean to the way pytorch engine handles them - request_stats contains more detailed info. and I'd like to have it returned by default with the iteration_stats to better understand what's happening during a specific iteration.
Any plans to rework the iteration_stats logic for trt engine? (if no, I'd be more than happy to raise a PR :D)
or am I misunderstanding something here? Is there a recommended way to link iteration_stats and request_stats?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the iteration and request statistics handling in tensorrt_llm/_torch/pyexecutor/py_executor.py around lines 714-717 and tensorrt_llm/executor/worker.py around lines 305-309. First determine how each backend currently associates these records, then decide whether the TRT response should expose request_stats with iteration_stats or provide another linkage. Done means the intended /metrics behavior is agreed and consistent for both backends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- api, backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100