aaif-goose / aaif-goose/goose

Todo extension: prompt-induced overhead, and whether it should stay a default

オープン
#11,172 コメント 3 件 リアクション 1 件 担当者 3 名 @DOsinga が担当を希望しています GitHub で見る
performance
主要言語
Rust
スター
54.2k
フォーク
6.2k
平均マージ
3日 4時間
マージ済み PR(30日)
240

説明

**What problem would this solve?**

The todo extension's prompts pressure models into checklist bookkeeping that raises cost without improving outcomes.

I benchmarked todo on/off as a single-variable A/B (Terminal-Bench 2.1, 26 tasks, k=3, stock goose) across four models:

| model | pass rate (off → on) | cost delta (on) |
|---|---|---|
| haiku-4.5 | 62.8% → 71.8% | +18% |
| sonnet-5 | 78.2% → 80.8% (noise) | +17% |
| opus-5 | 88.5% → 88.5% | +45% (median; means are outlier-dominated) |
| gpt-5.6-sol | 79.5% → 85.9% | +49% |

Three findings point at the prompts, not the tool:

1. The overhead is induced turns and per-turn context, not the tool calls or schema, so lazy-loading the tool would not recover it.
2. On hard tasks (FrontierBench, 10 tasks), todo improved nothing but sonnet-5 spent +39% on end-of-task "closure loops": re-verifying finished work to check off remaining items. The instructions say `End: verify all complete`.
3. Usage is near-universal (85-100% of trials) but uncorrelated with success. The empty-list turn reminder ("Once given a task, immediately update your todo with all explicit and implicit requirements") causes usage.

I then tested two variants on the models that benefit from todo:

| variant | gpt-5.6-sol | haiku-4.5 |
|---|---|---|
| stock todo on (baseline) | 85.9% at $0.165/trial | 65.4% at $0.188 |
| stock todo off (baseline) | 79.5% at $0.110 | 61.5% at $0.154 |
| V1: tool kept, prompts reframed as optional planning notes | 84.6% at $0.089 | 65.4% at $0.151 |
| V2: tool removed, one system-prompt line: brief plan first | 83.3% at $0.115 | 70.5% at $0.155 |

Both variants keep the pass-rate benefit at or below todo-off cost. Under V1, models nearly stop calling the tool while keeping the benefit. The instructions, not the tool, are the active ingredient. (k=3: treat pass-rate deltas as directional, the cost effects are consistent across all arms.)

**What would a good outcome look like?**

Two decisions, in increasing scope:

1. **Prompt fix (small, data-backed):** the todo instructions and turn reminder stop demanding completion and verification, and stop nagging when the list is empty. Tested language is available.
2. **Default status:** decide whether todo should remain a default extension. With pressure-free prompts the tool is rarely called, and a single planning line matches its measured benefit. Its remaining value is the interactive progress UX and a compaction-surviving scratchpad - weigh that against the extension surface (tool schema, per-turn context injection).

**Possible approaches**

- **Path A:** replace the instruction, tool-description, and turn-reminder strings in `crates/goose/src/agents/platform_extensions/todo.rs`. No behavior change otherwise.
- **Path B:** make todo opt-in and add the planning line to the system prompt. Needs desktop UX input, since the checklist is user-visible there.
- Path A stands alone and is worth doing even if Path B is rejected; if Path B proceeds, it supersedes A.

- [x] I have verified this does not duplicate an existing feature request

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

The issue is about the todo extension in `crates/goose/src/agents/platform_extensions/todo.rs`. Start by reading the current prompt strings and tool description in that file. Understand the benchmark results and the proposed prompt changes (Path A). If implementing Path A, modify the instruction, tool-description, and turn-reminder strings as suggested. Test the changes by running the agent to ensure the prompts no longer pressure the model into unnecessary checklist bookkeeping. A good outcome is the todo extension having pressure-free prompts while maintaining or improving cost efficiency.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
rust
領域
ai-infra-agents, tooling
issue の種類
機能追加
難易度
3/5
見積もり時間
1〜2日
活発さ
静か
明瞭さ
明確に書かれている
初心者へのやさしさ
45/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。