galaxyproject / galaxyproject/brc-analytics

Evals: multi-turn context-dependence set (gate for history-thinning)

Open
#1,369 0 comments 0 reactions 1 assignee Claimed by @NoopDog View on GitHub
enhancement
Dominant language
TypeScript
Stars
7
Forks
11
Avg merge
2d 12h
Merged PRs (30d)
16

Description

## Summary
Add a multi-turn eval set where each follow-up depends on a **prior** turn's tool output. It's the prerequisite for safely thinning the assistant's replayed history — without these cases we can't tell whether trimming history drops required context or just changes behavior harmlessly.

## Why
The assistant persists `result.all_messages()` and replays it verbatim every turn — tool calls **and** full tool-result JSON included (`assistant_agent.py:790`). We want to thin that (drop prior tool-call/return pairs to cut tokens), but the risk is breaking follow-ups that rely on earlier results. Need an objective before/after measure.

## What to add
Extend `evals/datasets/assistant_multiturn.py` (or a new `assistant_context.py`) with scripted conversations where turn N+1 needs turn N's tool output:
- **Reference / anaphora** — "complete-genome assemblies for *C. neoformans*" → "tell me about **the H99 one**" / "use **the second**."
- **Refinement chains** — "...for Anopheles" → "**narrow that** to chromosome level" → "**now sort by** N50."
- **Handoff to schema** — "...→ **set that assembly** for the workflow" (final `AnalysisSchema` check — the existing pattern).
- **Boundary / adversarial** — ask for a detail shown in the rows but **not** the prose summary (isolates "did dropping the result lose it?"); ask about a **never-returned** row (correct behavior = re-query / graceful, not hallucinate).

Scoring must separate *acceptable* from *degraded*:
- **Correctness + grounding** of the follow-up (deterministic value/keyword check where possible; LLM judge where not).
- **`tool_calls`** — re-running a tool to recover is acceptable; a wrong/ungrounded answer is not.
- **tokens** (already captured) so the cost/quality trade is visible.

## How it'll be used
Run this set under each history mode (full / last-turn-only / strip-all) when the thinning optimization lands; ship the most aggressive mode that holds correctness within threshold while cutting tokens. This ticket delivers the **cases + evaluators + a baseline** on current (full-history) behavior.

## Scope / non-goals
- This is the **eval set + evaluators**, not the history-thinning implementation (separate follow-on; this gates it).
- Reuse the existing harness (`run_evals.py`, `--repeat` for variance, model swap).

## Acceptance criteria
- [ ] Multi-turn cases covering reference, refinement, handoff, and boundary/adversarial.
- [ ] Evaluators score follow-up correctness + grounding and record `tool_calls` + tokens per case.
- [ ] Registered in `specs.py`; runnable via `python -m evals.run_evals --datasets `.
- [ ] Baseline scores captured on current full-history behavior (committed report).

## Context
**Post-beta enhancement under #1288 — not a beta blocker.** Gate for a future history-thinning optimization (the assistant replays full tool calls/results today, `assistant_agent.py:790`; prompt caching #1362 softens cost but not footprint).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.