aaif-goose / aaif-goose/goose

Todo extension: prompt-induced overhead, and whether it should stay a default

Ouverte
#11,172 3 commentaires 1 réaction 3 personnes assignées Réclamée par @DOsinga Voir sur GitHub
performance
Langage dominant
Rust
Étoiles
54.2k
Forks
6.2k
Merge moyen
3 j 4 h
PR mergées (30 j)
240

Description

**What problem would this solve?**

The todo extension's prompts pressure models into checklist bookkeeping that raises cost without improving outcomes.

I benchmarked todo on/off as a single-variable A/B (Terminal-Bench 2.1, 26 tasks, k=3, stock goose) across four models:

| model | pass rate (off → on) | cost delta (on) |
|---|---|---|
| haiku-4.5 | 62.8% → 71.8% | +18% |
| sonnet-5 | 78.2% → 80.8% (noise) | +17% |
| opus-5 | 88.5% → 88.5% | +45% (median; means are outlier-dominated) |
| gpt-5.6-sol | 79.5% → 85.9% | +49% |

Three findings point at the prompts, not the tool:

1. The overhead is induced turns and per-turn context, not the tool calls or schema, so lazy-loading the tool would not recover it.
2. On hard tasks (FrontierBench, 10 tasks), todo improved nothing but sonnet-5 spent +39% on end-of-task "closure loops": re-verifying finished work to check off remaining items. The instructions say `End: verify all complete`.
3. Usage is near-universal (85-100% of trials) but uncorrelated with success. The empty-list turn reminder ("Once given a task, immediately update your todo with all explicit and implicit requirements") causes usage.

I then tested two variants on the models that benefit from todo:

| variant | gpt-5.6-sol | haiku-4.5 |
|---|---|---|
| stock todo on (baseline) | 85.9% at $0.165/trial | 65.4% at $0.188 |
| stock todo off (baseline) | 79.5% at $0.110 | 61.5% at $0.154 |
| V1: tool kept, prompts reframed as optional planning notes | 84.6% at $0.089 | 65.4% at $0.151 |
| V2: tool removed, one system-prompt line: brief plan first | 83.3% at $0.115 | 70.5% at $0.155 |

Both variants keep the pass-rate benefit at or below todo-off cost. Under V1, models nearly stop calling the tool while keeping the benefit. The instructions, not the tool, are the active ingredient. (k=3: treat pass-rate deltas as directional, the cost effects are consistent across all arms.)

**What would a good outcome look like?**

Two decisions, in increasing scope:

1. **Prompt fix (small, data-backed):** the todo instructions and turn reminder stop demanding completion and verification, and stop nagging when the list is empty. Tested language is available.
2. **Default status:** decide whether todo should remain a default extension. With pressure-free prompts the tool is rarely called, and a single planning line matches its measured benefit. Its remaining value is the interactive progress UX and a compaction-surviving scratchpad - weigh that against the extension surface (tool schema, per-turn context injection).

**Possible approaches**

- **Path A:** replace the instruction, tool-description, and turn-reminder strings in `crates/goose/src/agents/platform_extensions/todo.rs`. No behavior change otherwise.
- **Path B:** make todo opt-in and add the planning line to the system prompt. Needs desktop UX input, since the checklist is user-visible there.
- Path A stands alone and is worth doing even if Path B is rejected; if Path B proceeds, it supersedes A.

- [x] I have verified this does not duplicate an existing feature request

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

L’issue concerne la todo extension dans `crates/goose/src/agents/platform_extensions/todo.rs`. Commencez par lire les chaînes de prompt actuelles et la description de l’outil dans ce fichier. Comprenez les résultats du benchmark et les modifications de prompt proposées (Path A). Si vous implémentez Path A, modifiez les chaînes Instruction, Tool-Description et Turn-Reminder comme suggéré. Testez les modifications en exécutant l’agent afin de vérifier que les prompts ne poussent plus le modèle à effectuer une gestion inutile de checklist. Un bon résultat serait que la todo extension dispose de prompts sans pression tout en maintenant ou en améliorant l’efficacité des coûts.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
rust
Domaine
ai-infra-agents, tooling
Type d'issue
Fonctionnalité
Difficulté
3/5
Temps estimé
1-2 jours
Activité
Calme
Clarté
Clairement spécifiée
Accessibilité débutants
45/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.