LMCache / LMCache/LMBenchmark

Issues with inf generation throughput and 0 ttft

Open
#27 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
61
Forks
26
PR merge metrics
No merged PRs in 30d

Description

# Summary

I've been working on a project involving lmcache tuning, and we have been using this benchmark as part of our evaluation suite.
The metric we are focusing on is "time to first token" (TTFT), and we have been noticing that many of the benchmarks are returning a time of **0** for this metric, which doesn't seem correct.

Additionally, we frequently see average throughput at `inf`

Is this working as designed? Could some guidance be offered how to better interpret these results?

# Details

Command invoked was `./run_benchmarks.sh "meta-llama/Llama-3.1-70B-Instruct" http://khanhtest-vllm-fs-lmcache.ibm-cas-red-stack.svc.cluster.local:8000 /tmp/benchmark/lmcache_off all 1.34 2.0 3.0`

```
/tmp/benchmark/lmcache_off_long_input_output_1.34.csv
[2025-06-16 11:44:01,006] WARNING: Processing the existing summary file /tmp/benchmark/lmcache_off_long_input_output_1.34.csv, ignoring all the other arguments (multi-round-qa.py:722:__main__)
[2025-06-16 11:44:01,008] INFO: Calculating performance summary (multi-round-qa.py:568:__main__)

==================== Performance summary ======================
QPS: 0.0000 reqs/s

Processing speed: 1.3941 reqs/s

Requests on-the-fly: 0

Input tokens per second: 29451.9382 tokens/s

Output tokens per second: 1.3941 tokens/s

Average generation throughput (per request): inf tokens/req/s
^^^^^^^^ inf??

Average TTFT: 0.0000s
^^^^^^^^^~ This takes no time at all?

Time range: 1750097945.6405346 - 1750098046.7831142 (101.14s)
===============================================================

```

[lmcache_off_long_input_output_1.34.csv](https://github.com/user-attachments/files/20763032/lmcache_off_long_input_output_1.34.csv)

Branch commit was `95b2939136ff003e2e4c67277ec82bf43ff8be34` on main.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing summary CSV and the performance-summary logic in multi-round-qa.py, using the run_benchmarks.sh command and the reported output as the reproduction path. Determine why average TTFT is 0 and generation throughput is inf, then ensure the benchmark output correctly reports or clearly explains these values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.