rossoctl / rossoctl/context-guru
cachesplit's -34.1% result is described as general, but it only applies to Claude Code CLI traffic
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 54
- Forks
- 22
- Avg merge
- 20h 42m
- Merged PRs (30d)
- 89
Description
What's wrong
config/config.go:369 describes cachesplit as:
It is also the best-evidenced single component in the repo: -34.1% cost and 0% ->
96.7% prefix-cache hit in an isolated A/B (docs/results/context-guru.md).
Three things about that are not right, and the repo already contains the corrections.
1. The cited doc is not an isolated cachesplit A/B. docs/results/context-guru.md
line 1 is titled "context-guru codesmart (final) (SWE-bench Verified, 50 tasks)" — a
whole-pipeline run. Nothing in it isolates the volatile-tail split.
2. The figure is workload- and CLIENT-specific, and two other docs say so.
docs/results/terminal-bench-comparison.md:155:
cachesplit acted=0for a structural reason: TB runs the Claude Agent SDK, which
never appends the git/env snapshot the CLI does, so all 73 captured requests carry 3
system blocks and zero volatile-tail markers. The -34.1% split figure came from
SWE-bench CLI traffic and does not transfer.
docs/results/improvement-plan.md:378 records the same thing as "zero legal opportunity".
3. It sits next to a much smaller production number. docs/results/measured-2026-08.md
credits the prefix split with $0.03 over the window (line 18), and line 179 explains it:
"1,799 mutations, credit granted on 3 requests". proxy/promexport.go:878 says to expect
it small on a warm-cache workload and calls the metric "A floor" — because the CLI
captures its environment block once per session, and 1,105 of 1,127 session starts were cold.
Why it matters
A reader deciding whether to run the component — or writing docs downstream of this comment —
takes "best-evidenced single component" and "-34.1%" as a property of the component. It is a
property of one workload on one client. The plugin's README and its plugin.json both lead
with the cache preset, so this comment is upstream of user-facing copy.
This is the defect shape already recorded twice on #249/#250: a measurement taken on one
workload, restated as a general property, surviving in every file nobody thought to grep.
Suggested fix
Documentation only — no behaviour change, and no argument here that the component should be
tuned or removed.
- In
config/config.go, name the workload and the client in the claim, drop "isolated A/B",
and point at the two docs that bound it. - State the production figure alongside it, so the comment carries both.
- Check whether the same sentence has been copied into
docs/reference/presets.mdor the
plugin'splugin.jsondescription.
What is NOT claimed here
The mechanism is sound where the shape exists: it removes no content, is byte-identical in
concatenation, fails open, and is a documented no-op on implicit-cache backends. This is
about the evidence sentence, not the code.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the cachesplit comment at config/config.go:369, then compare the cited evidence in docs/results/context-guru.md, docs/results/terminal-bench-comparison.md, docs/results/improvement-plan.md, and docs/results/measured-2026-08.md. Check docs/reference/presets.md and the plugin's plugin.json for copied wording; done means the documentation scopes the result to Claude Code CLI and the relevant workload, includes the production figure, and avoids implying a general component property.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100