rossoctl / rossoctl/context-guru

cachesplit's -34.1% result is described as general, but it only applies to Claude Code CLI traffic

Open Beginner friendly
#263 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation priority: low
Dominant language
Go
Stars
54
Forks
22
Avg merge
20h 42m
Merged PRs (30d)
89

Description

What's wrong

config/config.go:369 describes cachesplit as:

It is also the best-evidenced single component in the repo: -34.1% cost and 0% ->
96.7% prefix-cache hit in an isolated A/B (docs/results/context-guru.md).

Three things about that are not right, and the repo already contains the corrections.

1. The cited doc is not an isolated cachesplit A/B. docs/results/context-guru.md
line 1 is titled "context-guru codesmart (final) (SWE-bench Verified, 50 tasks)" — a
whole-pipeline run. Nothing in it isolates the volatile-tail split.

2. The figure is workload- and CLIENT-specific, and two other docs say so.

docs/results/terminal-bench-comparison.md:155:

cachesplit acted=0 for a structural reason: TB runs the Claude Agent SDK, which
never appends the git/env snapshot the CLI does, so all 73 captured requests carry 3
system blocks and zero volatile-tail markers. The -34.1% split figure came from
SWE-bench CLI traffic and does not transfer.

docs/results/improvement-plan.md:378 records the same thing as "zero legal opportunity".

3. It sits next to a much smaller production number. docs/results/measured-2026-08.md
credits the prefix split with $0.03 over the window (line 18), and line 179 explains it:
"1,799 mutations, credit granted on 3 requests". proxy/promexport.go:878 says to expect
it small on a warm-cache workload and calls the metric "A floor" — because the CLI
captures its environment block once per session, and 1,105 of 1,127 session starts were cold.

Why it matters

A reader deciding whether to run the component — or writing docs downstream of this comment —
takes "best-evidenced single component" and "-34.1%" as a property of the component. It is a
property of one workload on one client. The plugin's README and its plugin.json both lead
with the cache preset, so this comment is upstream of user-facing copy.

This is the defect shape already recorded twice on #249/#250: a measurement taken on one
workload, restated as a general property, surviving in every file nobody thought to grep.

Suggested fix

Documentation only — no behaviour change, and no argument here that the component should be
tuned or removed.

  • In config/config.go, name the workload and the client in the claim, drop "isolated A/B",
    and point at the two docs that bound it.
  • State the production figure alongside it, so the comment carries both.
  • Check whether the same sentence has been copied into docs/reference/presets.md or the
    plugin's plugin.json description.

What is NOT claimed here

The mechanism is sound where the shape exists: it removes no content, is byte-identical in
concatenation, fails open, and is a documented no-op on implicit-cache backends. This is
about the evidence sentence, not the code.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the cachesplit comment at config/config.go:369, then compare the cited evidence in docs/results/context-guru.md, docs/results/terminal-bench-comparison.md, docs/results/improvement-plan.md, and docs/results/measured-2026-08.md. Check docs/reference/presets.md and the plugin's plugin.json for copied wording; done means the documentation scopes the result to Claude Code CLI and the relevant workload, includes the production figure, and avoids implying a general component property.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.