Issues with inf generation throughput and 0 ttft
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 61
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
# Summary
I've been working on a project involving lmcache tuning, and we have been using this benchmark as part of our evaluation suite.
The metric we are focusing on is "time to first token" (TTFT), and we have been noticing that many of the benchmarks are returning a time of **0** for this metric, which doesn't seem correct.
Additionally, we frequently see average throughput at `inf`
Is this working as designed? Could some guidance be offered how to better interpret these results?
# Details
Command invoked was `./run_benchmarks.sh "meta-llama/Llama-3.1-70B-Instruct" http://khanhtest-vllm-fs-lmcache.ibm-cas-red-stack.svc.cluster.local:8000 /tmp/benchmark/lmcache_off all 1.34 2.0 3.0`
```
/tmp/benchmark/lmcache_off_long_input_output_1.34.csv
[2025-06-16 11:44:01,006] WARNING: Processing the existing summary file /tmp/benchmark/lmcache_off_long_input_output_1.34.csv, ignoring all the other arguments (multi-round-qa.py:722:__main__)
[2025-06-16 11:44:01,008] INFO: Calculating performance summary (multi-round-qa.py:568:__main__)
==================== Performance summary ======================
QPS: 0.0000 reqs/s
Processing speed: 1.3941 reqs/s
Requests on-the-fly: 0
Input tokens per second: 29451.9382 tokens/s
Output tokens per second: 1.3941 tokens/s
Average generation throughput (per request): inf tokens/req/s
^^^^^^^^ inf??
Average TTFT: 0.0000s
^^^^^^^^^~ This takes no time at all?
Time range: 1750097945.6405346 - 1750098046.7831142 (101.14s)
===============================================================
```
[lmcache_off_long_input_output_1.34.csv](https://github.com/user-attachments/files/20763032/lmcache_off_long_input_output_1.34.csv)
Branch commit was `95b2939136ff003e2e4c67277ec82bf43ff8be34` on main.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing summary CSV and the performance-summary logic in multi-round-qa.py, using the run_benchmarks.sh command and the reported output as the reproduction path. Determine why average TTFT is 0 and generation throughput is inf, then ensure the benchmark output correctly reports or clearly explains these values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100