Todo extension: prompt-induced overhead, and whether it should stay a default
- Ngôn ngữ chính
- Rust
- Star
- 54.2k
- Fork
- 6.2k
- Merge trung bình
- 3 ngày 4 giờ
- Pull request đã merge (30 ngày)
- 240
Mô tả
**What problem would this solve?**
The todo extension's prompts pressure models into checklist bookkeeping that raises cost without improving outcomes.
I benchmarked todo on/off as a single-variable A/B (Terminal-Bench 2.1, 26 tasks, k=3, stock goose) across four models:
| model | pass rate (off → on) | cost delta (on) |
|---|---|---|
| haiku-4.5 | 62.8% → 71.8% | +18% |
| sonnet-5 | 78.2% → 80.8% (noise) | +17% |
| opus-5 | 88.5% → 88.5% | +45% (median; means are outlier-dominated) |
| gpt-5.6-sol | 79.5% → 85.9% | +49% |
Three findings point at the prompts, not the tool:
1. The overhead is induced turns and per-turn context, not the tool calls or schema, so lazy-loading the tool would not recover it.
2. On hard tasks (FrontierBench, 10 tasks), todo improved nothing but sonnet-5 spent +39% on end-of-task "closure loops": re-verifying finished work to check off remaining items. The instructions say `End: verify all complete`.
3. Usage is near-universal (85-100% of trials) but uncorrelated with success. The empty-list turn reminder ("Once given a task, immediately update your todo with all explicit and implicit requirements") causes usage.
I then tested two variants on the models that benefit from todo:
| variant | gpt-5.6-sol | haiku-4.5 |
|---|---|---|
| stock todo on (baseline) | 85.9% at $0.165/trial | 65.4% at $0.188 |
| stock todo off (baseline) | 79.5% at $0.110 | 61.5% at $0.154 |
| V1: tool kept, prompts reframed as optional planning notes | 84.6% at $0.089 | 65.4% at $0.151 |
| V2: tool removed, one system-prompt line: brief plan first | 83.3% at $0.115 | 70.5% at $0.155 |
Both variants keep the pass-rate benefit at or below todo-off cost. Under V1, models nearly stop calling the tool while keeping the benefit. The instructions, not the tool, are the active ingredient. (k=3: treat pass-rate deltas as directional, the cost effects are consistent across all arms.)
**What would a good outcome look like?**
Two decisions, in increasing scope:
1. **Prompt fix (small, data-backed):** the todo instructions and turn reminder stop demanding completion and verification, and stop nagging when the list is empty. Tested language is available.
2. **Default status:** decide whether todo should remain a default extension. With pressure-free prompts the tool is rarely called, and a single planning line matches its measured benefit. Its remaining value is the interactive progress UX and a compaction-surviving scratchpad - weigh that against the extension surface (tool schema, per-turn context injection).
**Possible approaches**
- **Path A:** replace the instruction, tool-description, and turn-reminder strings in `crates/goose/src/agents/platform_extensions/todo.rs`. No behavior change otherwise.
- **Path B:** make todo opt-in and add the planning line to the system prompt. Needs desktop UX input, since the checklist is user-visible there.
- Path A stands alone and is worth doing even if Path B is rejected; if Path B proceeds, it supersedes A.
- [x] I have verified this does not duplicate an existing feature request
Hướng dẫn đóng góp
Hướng nghiên cứu
The issue is about the todo extension in `crates/goose/src/agents/platform_extensions/todo.rs`. Start by reading the current prompt strings and tool description in that file. Understand the benchmark results and the proposed prompt changes (Path A). If implementing Path A, modify the instruction, tool-description, and turn-reminder strings as suggested. Test the changes by running the agent to ensure the prompts no longer pressure the model into unnecessary checklist bookkeeping. A good outcome is the todo extension having pressure-free prompts while maintaining or improving cost efficiency.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- rust
- Lĩnh vực
- ai-infra-agents, tooling
- Loại issue
- Tính năng
- Độ khó
- 3/5
- Thời gian dự kiến
- 1-2 ngày
- Mức độ hoạt động
- Ít trao đổi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 45/100