Feature request: KV-cache aware compaction to eliminate post-compaction cache miss

Feature request: KV-cache aware compaction to eliminate post-compaction cache miss

Context

Kimi Code CLI 0.37.1 triggers automatic context compaction when token pressure exceeds an internal threshold. This compaction is effective at reducing context size (−81% to −83% input reduction at the first post-compaction call), but the first LLM call after compaction systematically misses the KV-cache: the system prompt, tool schemas, and compacted summary — which are identical or near-identical to the previous call — are re-tokenized and re-billed as fresh input.

The wire.jsonl session log contains a context.apply_compaction marker with tokensBefore/tokensAfter fields. A local measurement probe (compaction-cost-probe.py) extracts these markers from historical sessions without consuming tokens. Measured across 60 sessions on the principal workdir (internal workdir hash omitted):

Session	Model	input before	input after	delta	Reduction
0a6fa689 (2026-07-05)	kimi-for-coding	212,039	36,023	−176,016	−83%
01e76c29 (2026-07-12)	kimi-for-coding	211,670	39,544	−172,126	−81%
da054a46 (2026-08-13)	k3-256k	212,154	41,250	−170,904	−81%

For session 0a6fa689, inputCacheRead = 0 on the first post-compaction call — the ~36k tokens were billed entirely as fresh input despite the system prompt and tool schemas being identical to the previous call. For 01e76c29, inputCacheRead = 34,560 (partially cached). This inconsistency means the KV-cache is not reliably reused after compaction.

Problem or Gap

After compaction, Kimi Code CLI reconstructs a new request prefix that can diverge from the previous request (ordering, summary markers, token counts). This divergence invalidates the provider's prefix cache, forcing a full re-tokenization of the system prompt and tool schemas at the first post-compaction call.

The context.apply_compaction marker exists in the wire.jsonl but is neither documented nor stable across releases. There is no hook event emitted at compaction time (the PreCompact/PostCompact hooks appear in the binary but are not documented or exposed by default in config.toml). The compaction summary is opaque — no log-only audit trail, no configurable parameters, no attribution in usage metrics.

Proposal

U1 — Verbatim prefix replay for KV-cache reuse

The compaction summary call must be constructed as an exact prefix of the last routed request: same system prompt, same tool schemas, same retained messages, with the compaction instruction appended as the final user message. This ensures the provider's prefix cache is reused and only the compaction instruction is billed as new input.

U2 — Log-only audit events

Emit compaction/start, compaction/summary, compaction/end, and compaction/error events in the session journal, without injecting them into the model-visible context. The existing context.apply_compaction marker should be documented and stabilized.

U3 — Configurable parameters

Expose in configuration (not hardcoded in the binary):

• Compaction trigger threshold (token pressure ratio)
• Recent queue retention (number of recent messages/turns kept verbatim)
• Summary instruction (customizable)
• Checkpoint marking tags (e.g., <compacted-summary>)

U4 — Summary call attribution

Distinguish the compaction summary call from normal calls in metrics and billing via a dedicated purpose field or header (e.g., purpose: "compaction"). This enables separate cost tracking for compaction overhead.

U5 — Text-only checkpoint

Store only the text returned by the summary in the checkpoint: exclude reasoning and tool calls from the compacted content. Mark with explicit tags and a fixed preamble.

Impact

• Cost reduction: Eliminating the post-compaction cache miss saves ~36k–41k tokens of redundant input billing per compaction event. At 60+ compaction events observed in a single workdir, this represents significant cumulative savings.
• Auditability: Log-only events and a documented context.apply_compaction format enable reliable forensics without depending on undocumented internals.
• Tunability: Configurable thresholds and retention allow users to balance compaction aggressiveness against context continuity.
• Measurement: Summary call attribution enables precise cost accounting for compaction vs. user work.

Acceptance criteria

1. The first post-compaction call reuses the prefix cache: inputCacheRead covers the system prompt + tool schemas (unchanged). Verified by the measurement probe on a test session.
2. A compaction/start|summary|end event is present in the session journal for each compaction, and is NOT injected into the model context.
3. Trigger threshold and queue retention are configurable and effective (changing config changes compaction behavior).
4. The summary call is attributed (purpose=compaction) and separable in metrics.
5. The checkpoint contains only the summary text (no reasoning, no tool calls).
6. Input reduction at the first post-compaction call remains ≥ 80% on a standard long session (reference baseline: −81% to −83%).

Start with the context.apply_compaction marker in wire.jsonl and the compaction-cost-probe.py measurement probe. Review the mentioned config.toml settings and PreCompact/PostCompact hooks before assessing the proposal. Done means meeting the six acceptance criteria, including cache reuse, journal events, configurable behavior, attribution, text-only checkpoints, and at least 80% input reduction.