Todo extension: prompt-induced overhead, and whether it should stay a default
- 主要言語
- Rust
- スター
- 54.2k
- フォーク
- 6.2k
- 平均マージ
- 3日 4時間
- マージ済み PR(30日)
- 240
説明
**What problem would this solve?**
The todo extension's prompts pressure models into checklist bookkeeping that raises cost without improving outcomes.
I benchmarked todo on/off as a single-variable A/B (Terminal-Bench 2.1, 26 tasks, k=3, stock goose) across four models:
| model | pass rate (off → on) | cost delta (on) |
|---|---|---|
| haiku-4.5 | 62.8% → 71.8% | +18% |
| sonnet-5 | 78.2% → 80.8% (noise) | +17% |
| opus-5 | 88.5% → 88.5% | +45% (median; means are outlier-dominated) |
| gpt-5.6-sol | 79.5% → 85.9% | +49% |
Three findings point at the prompts, not the tool:
1. The overhead is induced turns and per-turn context, not the tool calls or schema, so lazy-loading the tool would not recover it.
2. On hard tasks (FrontierBench, 10 tasks), todo improved nothing but sonnet-5 spent +39% on end-of-task "closure loops": re-verifying finished work to check off remaining items. The instructions say `End: verify all complete`.
3. Usage is near-universal (85-100% of trials) but uncorrelated with success. The empty-list turn reminder ("Once given a task, immediately update your todo with all explicit and implicit requirements") causes usage.
I then tested two variants on the models that benefit from todo:
| variant | gpt-5.6-sol | haiku-4.5 |
|---|---|---|
| stock todo on (baseline) | 85.9% at $0.165/trial | 65.4% at $0.188 |
| stock todo off (baseline) | 79.5% at $0.110 | 61.5% at $0.154 |
| V1: tool kept, prompts reframed as optional planning notes | 84.6% at $0.089 | 65.4% at $0.151 |
| V2: tool removed, one system-prompt line: brief plan first | 83.3% at $0.115 | 70.5% at $0.155 |
Both variants keep the pass-rate benefit at or below todo-off cost. Under V1, models nearly stop calling the tool while keeping the benefit. The instructions, not the tool, are the active ingredient. (k=3: treat pass-rate deltas as directional, the cost effects are consistent across all arms.)
**What would a good outcome look like?**
Two decisions, in increasing scope:
1. **Prompt fix (small, data-backed):** the todo instructions and turn reminder stop demanding completion and verification, and stop nagging when the list is empty. Tested language is available.
2. **Default status:** decide whether todo should remain a default extension. With pressure-free prompts the tool is rarely called, and a single planning line matches its measured benefit. Its remaining value is the interactive progress UX and a compaction-surviving scratchpad - weigh that against the extension surface (tool schema, per-turn context injection).
**Possible approaches**
- **Path A:** replace the instruction, tool-description, and turn-reminder strings in `crates/goose/src/agents/platform_extensions/todo.rs`. No behavior change otherwise.
- **Path B:** make todo opt-in and add the planning line to the system prompt. Needs desktop UX input, since the checklist is user-visible there.
- Path A stands alone and is worth doing even if Path B is rejected; if Path B proceeds, it supersedes A.
- [x] I have verified this does not duplicate an existing feature request
コントリビューションガイド
調査の方向性
The issue is about the todo extension in `crates/goose/src/agents/platform_extensions/todo.rs`. Start by reading the current prompt strings and tool description in that file. Understand the benchmark results and the proposed prompt changes (Path A). If implementing Path A, modify the instruction, tool-description, and turn-reminder strings as suggested. Test the changes by running the agent to ensure the prompts no longer pressure the model into unnecessary checklist bookkeeping. A good outcome is the todo extension having pressure-free prompts while maintaining or improving cost efficiency.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- rust
- 領域
- ai-infra-agents, tooling
- issue の種類
- 機能追加
- 難易度
- 3/5
- 見積もり時間
- 1〜2日
- 活発さ
- 静か
- 明瞭さ
- 明確に書かれている
- 初心者へのやさしさ
- 45/100