kvcache-ai / kvcache-ai/Mooncake

[RFC]: Reproducing Table 1 hit rates on the released trace, consistent +4-6pp offset

Open
#4,020 1 comment 0 reactions 1 assignee Claimed by @chestnut-Q View on GitHub
RFC
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Changes proposed

I've been reproducing your Table 1 cache hit rates on arxiv-trace/mooncake_trace.jsonl using LRU, and I get the same curve shape but a systematic offset at every capacity:

| blocks | 1k | 10k | 30k | 50k | 100k | infiniti |
|--------|-------|-------|-------|-------|-------|-------|
| paper | 0.30 | 0.40 | 0.48 | 0.50 | 0.51 | 0.51 |
| mine | 0.341 | 0.460 | 0.537 | 0.551 | 0.552 | 0.553 |

I compute hit rate as (sum of longest-resident-prefix blocks) / (sum of all blocks requested), replaying in timestamp order. Since the infinite-cache case is policy-free, I'd expect it to be reproducible from the trace alone, but I get 0.553 against your 0.51.

I've tried five denominator definitions (blocks vs input_length tokens, excluding the partial tail block, per-request averaging) and none closes the gap. Is the released trace the same one used for Table 1, or is there a detail in the hit-rate definition I'm missing?

Repro: https://github.com/gauravapiscean/agentic-kv-cache

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.