galaxyproject / galaxyproject/brc-analytics
Assistant: thin replayed history (drop prior tool-call/result pairs) to cut tokens
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 11
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 16
Description
## Summary
Cut token/cost growth in multi-turn conversations by **thinning the replayed message history** — drop prior turns' tool-call/return pairs, keeping user + assistant text. Today the assistant replays `result.all_messages()` verbatim (full tool calls + result JSON) every turn (`assistant_agent.py:790`), so tool-result payloads accumulate across the conversation.
## Depends on
- **#1369** (multi-turn context-dependence eval set) — **required gate.** Do not ship a thinning mode without A/B-measuring it against #1369: correctness + grounding must hold; re-running a tool is acceptable, a wrong/ungrounded answer is not.
- Related: **#1362** (prompt caching — softens cost but not footprint).
## What to do
- Add a **history-mode** knob (setting): `full` (current) / `last-turn-only` (keep only the most recent turn's tool parts) / `strip-all` (drop all tool-call/return pairs; replay user + assistant text only).
- Apply the transform where history is restored/replayed (`assistant_agent.py:726–733`, before `agent.run(..., message_history=…)`). Keep history **well-formed**: drop a tool-call and its tool-return **together** (pydantic-ai requires the pair) — never orphan a call.
- Default to `full` until the eval A/B picks a winner, then flip the default.
## How the mode is chosen (the gate)
Run #1369's set under each mode; compare correctness, grounding, `tool_calls`, tokens. Ship the **most aggressive mode that holds correctness within threshold** while cutting tokens. `last-turn-only` is the safe fallback if `strip-all` costs too much re-querying.
## Scope / non-goals
- History replay only — does not change tool behavior, schema state, or the existing `_truncate_history` count cap (kept as a backstop).
- No summarization variant here (possible later option if `strip-all` degrades).
## Acceptance criteria
- [ ] History-mode knob (`full` / `last-turn-only` / `strip-all`); replayed history stays well-formed (paired tool parts).
- [ ] A/B results on #1369 recorded; chosen default justified by the numbers (token cut vs. correctness).
- [ ] Token reduction demonstrated on a multi-turn probe; no correctness/grounding regression on #1369 at the chosen mode.
## Context
Post-beta enhancement under #1288 — not a beta blocker. Pairs with #1369 (its gate).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.