galaxyproject / galaxyproject/brc-analytics
Evals: multi-turn context-dependence set (gate for history-thinning)
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 11
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 16
Description
## Summary
Add a multi-turn eval set where each follow-up depends on a **prior** turn's tool output. It's the prerequisite for safely thinning the assistant's replayed history — without these cases we can't tell whether trimming history drops required context or just changes behavior harmlessly.
## Why
The assistant persists `result.all_messages()` and replays it verbatim every turn — tool calls **and** full tool-result JSON included (`assistant_agent.py:790`). We want to thin that (drop prior tool-call/return pairs to cut tokens), but the risk is breaking follow-ups that rely on earlier results. Need an objective before/after measure.
## What to add
Extend `evals/datasets/assistant_multiturn.py` (or a new `assistant_context.py`) with scripted conversations where turn N+1 needs turn N's tool output:
- **Reference / anaphora** — "complete-genome assemblies for *C. neoformans*" → "tell me about **the H99 one**" / "use **the second**."
- **Refinement chains** — "...for Anopheles" → "**narrow that** to chromosome level" → "**now sort by** N50."
- **Handoff to schema** — "...→ **set that assembly** for the workflow" (final `AnalysisSchema` check — the existing pattern).
- **Boundary / adversarial** — ask for a detail shown in the rows but **not** the prose summary (isolates "did dropping the result lose it?"); ask about a **never-returned** row (correct behavior = re-query / graceful, not hallucinate).
Scoring must separate *acceptable* from *degraded*:
- **Correctness + grounding** of the follow-up (deterministic value/keyword check where possible; LLM judge where not).
- **`tool_calls`** — re-running a tool to recover is acceptable; a wrong/ungrounded answer is not.
- **tokens** (already captured) so the cost/quality trade is visible.
## How it'll be used
Run this set under each history mode (full / last-turn-only / strip-all) when the thinning optimization lands; ship the most aggressive mode that holds correctness within threshold while cutting tokens. This ticket delivers the **cases + evaluators + a baseline** on current (full-history) behavior.
## Scope / non-goals
- This is the **eval set + evaluators**, not the history-thinning implementation (separate follow-on; this gates it).
- Reuse the existing harness (`run_evals.py`, `--repeat` for variance, model swap).
## Acceptance criteria
- [ ] Multi-turn cases covering reference, refinement, handoff, and boundary/adversarial.
- [ ] Evaluators score follow-up correctness + grounding and record `tool_calls` + tokens per case.
- [ ] Registered in `specs.py`; runnable via `python -m evals.run_evals --datasets `.
- [ ] Baseline scores captured on current full-history behavior (committed report).
## Context
**Post-beta enhancement under #1288 — not a beta blocker.** Gate for a future history-thinning optimization (the assistant replays full tool calls/results today, `assistant_agent.py:790`; prompt caching #1362 softens cost but not footprint).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.