mlcommons / mlcommons/storage

KVCache Verify client CPU choice has minimal influence on results

Open
#334 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Future KVCache TF
Dominant language
Python
Stars
203
Forks
67
Avg merge
20m
Merged PRs (30d)
8

Description

One of the goals of the MLPerf Storage benchmarks are that once the storage device is saturated (peak throughput/IOPS), adding additional compute resources to a test client does not significantly alter the reported metrics. Specifically, our benchmarks should be storage benchmarks, not server benchmarks.

For each of the four KV Cache metrics reported on the results table, we need to verify that this above property is true:

  1. ⚠️ Tokens per second - If locally computed tokens (CPU-driven) are included together with KV cache'd tokens, having a faster client CPU would increase this metric. As a result, we should only include "Cached tokens per second". Since our KV cache workload is fixed across submitters for closed submissions, this will allow for a fair comparison.

  2. ✅ Read bandwidth - Once the storage is saturated, adding additional client CPU resources will not further increase this number.

  3. ✅ Write bandwidth - Once the storage is saturated, adding additional client CPU resources will not further increase this number.

  4. ⚠️ P95 Read Latency - If local computation steps are included in this latency measurement, having a faster client CPU would improve this metric. As a result, we should ensure that we are only including the latency of the storage I/O operations in this metric.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the KV Cache workload code and the results-table calculations for tokens per second and P95 read latency. Trace whether local CPU computation is included in those metrics, while checking the existing read and write bandwidth behavior. Done means tokens per second reports only cached tokens and P95 read latency covers only storage I/O, without changing the bandwidth metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.