KVCache Verify client CPU choice has minimal influence on results
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 203
- Forks
- 67
- Avg merge
- 20m
- Merged PRs (30d)
- 8
Description
One of the goals of the MLPerf Storage benchmarks are that once the storage device is saturated (peak throughput/IOPS), adding additional compute resources to a test client does not significantly alter the reported metrics. Specifically, our benchmarks should be storage benchmarks, not server benchmarks.
For each of the four KV Cache metrics reported on the results table, we need to verify that this above property is true:
-
⚠️ Tokens per second - If locally computed tokens (CPU-driven) are included together with KV cache'd tokens, having a faster client CPU would increase this metric. As a result, we should only include "Cached tokens per second". Since our KV cache workload is fixed across submitters for closed submissions, this will allow for a fair comparison.
-
✅ Read bandwidth - Once the storage is saturated, adding additional client CPU resources will not further increase this number.
-
✅ Write bandwidth - Once the storage is saturated, adding additional client CPU resources will not further increase this number.
-
⚠️ P95 Read Latency - If local computation steps are included in this latency measurement, having a faster client CPU would improve this metric. As a result, we should ensure that we are only including the latency of the storage I/O operations in this metric.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the KV Cache workload code and the results-table calculations for tokens per second and P95 read latency. Trace whether local CPU computation is included in those metrics, while checking the existing read and write bandwidth behavior. Done means tokens per second reports only cached tokens and P95 read latency covers only storage I/O, without changing the bandwidth metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100