aaif-goose / aaif-goose/goose

Todo extension: prompt-induced overhead, and whether it should stay a default

Offen
#11,172 3 Kommentare 1 Reaktion 3 zugewiesene Personen Beansprucht von @DOsinga Auf GitHub ansehen
performance
Vorherrschende Sprache
Rust
Sterne
54.2k
Forks
6.2k
Ø Merge
3 T. 2 Std.
Gemergte PRs (30 T.)
262

Beschreibung

**What problem would this solve?**

The todo extension's prompts pressure models into checklist bookkeeping that raises cost without improving outcomes.

I benchmarked todo on/off as a single-variable A/B (Terminal-Bench 2.1, 26 tasks, k=3, stock goose) across four models:

| model | pass rate (off → on) | cost delta (on) |
|---|---|---|
| haiku-4.5 | 62.8% → 71.8% | +18% |
| sonnet-5 | 78.2% → 80.8% (noise) | +17% |
| opus-5 | 88.5% → 88.5% | +45% (median; means are outlier-dominated) |
| gpt-5.6-sol | 79.5% → 85.9% | +49% |

Three findings point at the prompts, not the tool:

1. The overhead is induced turns and per-turn context, not the tool calls or schema, so lazy-loading the tool would not recover it.
2. On hard tasks (FrontierBench, 10 tasks), todo improved nothing but sonnet-5 spent +39% on end-of-task "closure loops": re-verifying finished work to check off remaining items. The instructions say `End: verify all complete`.
3. Usage is near-universal (85-100% of trials) but uncorrelated with success. The empty-list turn reminder ("Once given a task, immediately update your todo with all explicit and implicit requirements") causes usage.

I then tested two variants on the models that benefit from todo:

| variant | gpt-5.6-sol | haiku-4.5 |
|---|---|---|
| stock todo on (baseline) | 85.9% at $0.165/trial | 65.4% at $0.188 |
| stock todo off (baseline) | 79.5% at $0.110 | 61.5% at $0.154 |
| V1: tool kept, prompts reframed as optional planning notes | 84.6% at $0.089 | 65.4% at $0.151 |
| V2: tool removed, one system-prompt line: brief plan first | 83.3% at $0.115 | 70.5% at $0.155 |

Both variants keep the pass-rate benefit at or below todo-off cost. Under V1, models nearly stop calling the tool while keeping the benefit. The instructions, not the tool, are the active ingredient. (k=3: treat pass-rate deltas as directional, the cost effects are consistent across all arms.)

**What would a good outcome look like?**

Two decisions, in increasing scope:

1. **Prompt fix (small, data-backed):** the todo instructions and turn reminder stop demanding completion and verification, and stop nagging when the list is empty. Tested language is available.
2. **Default status:** decide whether todo should remain a default extension. With pressure-free prompts the tool is rarely called, and a single planning line matches its measured benefit. Its remaining value is the interactive progress UX and a compaction-surviving scratchpad - weigh that against the extension surface (tool schema, per-turn context injection).

**Possible approaches**

- **Path A:** replace the instruction, tool-description, and turn-reminder strings in `crates/goose/src/agents/platform_extensions/todo.rs`. No behavior change otherwise.
- **Path B:** make todo opt-in and add the planning line to the system prompt. Needs desktop UX input, since the checklist is user-visible there.
- Path A stands alone and is worth doing even if Path B is rejected; if Path B proceeds, it supersedes A.

- [x] I have verified this does not duplicate an existing feature request

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.