folder-QA v2: qualitative/research-style questions — and the grading problem they create

Sub-thread of #68, flagged as the biggest lift of the three v2 tiers because it needs new grading infrastructure, not just new cases.

**What's missing.** Every case in the battery today (v1 and the two sibling v2 issues) is ultimately a numeric/categorical/text lookup with one deterministic gold. Real users also ask genuinely open-ended, qualitative questions that have no single correct answer:

- **Synthesis over free text** — "summarize what happened in my journal this year," "what patterns do you notice in my spending habits," "what's changed about my routine since spring."
- **Cross-signal reasoning that isn't a single aggregate** — "did my mood track with how much I worked out," "what tends to happen around my big spending months" — combining a qualitative signal (journal text) with a quantitative one (workouts/spend) in a way that doesn't reduce to one SQL query.
- **Narrative/holistic requests** — "tell me about my year," "how am I doing financially, overall."

**Why this needs its own design work, not just cases.** `gen.py`'s gold types (`figures`/`approx`/`contains`/`refusal`/`notool`) assume an exact or substring-matchable answer. A synthesis answer has no single correct string. Options, roughly in order of effort:

1. **"Must mention K of N key facts"** — extend `contains` (or a new gold type) to partial credit: gold lists N facts the answer should touch (e.g. 3 of 5 specific journal themes, or the correct direction + rough magnitude of a trend), scored as a fraction rather than all-or-nothing.
2. **Rubric-based LLM-judge grading** — `agent_eval` already has `--judge ` support; define a per-case rubric (what a good answer covers, what it must NOT claim) and have the judge score against it. Needs judge-consistency validation (does the judge agree with itself / a human across a re-run) before it's trusted as a metric, the same rigor the deterministic golds got.
3. **Periodic human spot-check** — not automated at all, just a manual audit on a sample each time the loop changes. Cheapest, least scalable, best as a stopgap while (1)/(2) get built.

**Recommendation:** prototype small (5-10 cases) with option 1 first since it stays deterministic and fits the existing tooling with the least new infrastructure; treat option 2 as the real long-term answer once there's a rubric-writing pattern worth standardizing.

Related: #68 (parent), #84 (surfacing low confidence) — a synthesis answer with no crisp gold is exactly the kind of case where visible confidence-signaling matters most.

Start with gen.py's existing gold types and agent_eval's --judge path, then review the current deterministic cases and grading tests. Prototype 5–10 qualitative cases using partial-credit key facts, and define how scores and missing facts are reported; done means the prototype runs through the existing evaluation flow and its scoring is validated against the expected facts.

Written by the indexing model from the issue text.