Cache design: content-addressed local SQLite store

Context

Follow-up to #36. The langchain cache does not work with vLLM and gives little control, and the earlier drafts mixed identity keys with content keys in different ways. There is now a lean draft stack on top of #118 (#121–#124, #127) that tries to settle the local cache design. I would like to walk through the current approach and get feedback.

What we do

We cache both completions and judgements in separate SQLite trees under an optional --run.store_root. The cache boundary is do_inference. The model is prepared lazily, so a full hit does not load the backend.

Rows are keyed by a hash of the rendered input, and live under a folder keyed by the validated model descriptor (sampling, max tokens, thinking settings, chat template, output-affecting engine settings). Completions and judgements stay in separate role roots so the same model string as battle model and as judge cannot collide and thus they have separate metadata files. Typed columns (benchmark, instruction_id, model / model_a/model_b, orientation, …) are kept for inspection and targeted deletion, but we don't use them as key anymore. Prompt-construction options whose effect already appears in the rendered messages are captured by the input hash. Parsing and aggregation stay outside the raw-output cache.

We additionally add a Hugging Face fetch/push as a separate CLI in #127.

Layout

{store_root}/completions/{task}/{provider}/{model}/{descriptor_hash}/
├── metadata.json
└── completions.db

{store_root}/judgements/{task}/{provider}/{model}/{descriptor_hash}/
├── metadata.json
└── judgements.db

metadata.json stores the descriptor.

Schema

completions(
  input_hash TEXT PRIMARY KEY,
  input_text TEXT NOT NULL,
  completion TEXT NOT NULL,
  benchmark TEXT NOT NULL,
  instruction_id TEXT NOT NULL,
  model TEXT NOT NULL,
  pushed_by TEXT NOT NULL,
  pushed_at TEXT NOT NULL,
  run_id TEXT NOT NULL
)

judgements(
  input_hash TEXT PRIMARY KEY,
  judge_input TEXT NOT NULL,
  judge_completion TEXT NOT NULL,
  benchmark TEXT NOT NULL,
  instruction_id TEXT NOT NULL,
  model_a TEXT NOT NULL,
  model_b TEXT NOT NULL,
  judge TEXT NOT NULL,
  top_logprobs TEXT,
  orientation TEXT,
  pushed_by TEXT NOT NULL,
  pushed_at TEXT NOT NULL,
  run_id TEXT NOT NULL
)

top_logprobs is there so cached judgements can restore the first-token logprobs used by the #118 parsers. orientation is direct/reversed relative to the source model order when that applies. model_a/model_b match the positional slots in the rendered judge prompt.

Why this shape

Identity keys alone miss cases where the same battle is judged with a different rendered prompt or different model settings. Content keys alone don't provide easy filtering of the rows. The current shape keeps content-addressed hits, and the other fields for filtering and targeted deletion of the cache. VLLM execution-only settings (tensor_parallel_size, gpu_memory_utilization, enforce_eager) stay out of the descriptor.

Any thoughts on this? cc: @kargibora

Start by reading the current draft stack referenced in #118, #121–#124, and #127, with particular attention to the do_inference cache boundary and the proposed SQLite layout. The issue asks for design feedback rather than defining a concrete implementation or completion test, so the done criteria would need to be settled before coding begins.