galaxyproject / galaxyproject/loom
Weaker models (GPT-4o) skip notebook.md logging even when steered; stronger models comply
- Dominant language
- TypeScript
- Stars
- 14
- Forks
- 12
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 17
Description
### Observed
The same task (clone a GitHub repo, set up a conda/mamba env, install per the README) was run under two models. **GPT-4o** carried out the shell commands but **never updated `notebook.md`** to log its attempts or outcomes. **gemini-flash-latest** completed the task *and* correctly updated `notebook.md` automatically. The tester is fine with the install failing; the not-fine part is the missing notebook logging.
### Evidence
In the activity log under gpt-4o, the tail is all `bash` tool calls (conda/mamba failures) and **zero `edit`/`write` against notebook.md**. Once the session switches to gemini-flash-latest, `read(notebook.md)` then `write(notebook.md)` appear immediately. The notebook-maintenance instruction in the system prompt (`extensions/loom/context.ts` "Notebook (project log)" block) is honored by some models and ignored by GPT-4o, especially on the failure path (it logged nothing precisely when commands failed).
### Distinct from related issues
- Not #253 (panel auto-refresh) -- the file is genuinely *not written* here, not "written but not displayed."
- Distinct from the display-refresh angle reported separately for gpt-5.5.
### Fix direction (brain-guidance)
Strengthen notebook-logging steering for weaker models -- e.g. an end-of-turn nudge or deterministic reminder to log attempts/outcomes to notebook.md, especially after failed commands. Fits #99 (right-size prompt for smaller models). A deterministic notebook-append hook for key session events would make logging less dependent on model compliance.
### Environment
Orbit 0.4.1, Linux x64; failing model openai/gpt-4o, working model google/gemini-flash-latest.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.