galaxyproject / galaxyproject/loom

Weaker models (GPT-4o) skip notebook.md logging even when steered; stronger models comply

Open
#317 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
14
Forks
12
Avg merge
6d 5h
Merged PRs (30d)
17

Description

### Observed
The same task (clone a GitHub repo, set up a conda/mamba env, install per the README) was run under two models. **GPT-4o** carried out the shell commands but **never updated `notebook.md`** to log its attempts or outcomes. **gemini-flash-latest** completed the task *and* correctly updated `notebook.md` automatically. The tester is fine with the install failing; the not-fine part is the missing notebook logging.

### Evidence
In the activity log under gpt-4o, the tail is all `bash` tool calls (conda/mamba failures) and **zero `edit`/`write` against notebook.md**. Once the session switches to gemini-flash-latest, `read(notebook.md)` then `write(notebook.md)` appear immediately. The notebook-maintenance instruction in the system prompt (`extensions/loom/context.ts` "Notebook (project log)" block) is honored by some models and ignored by GPT-4o, especially on the failure path (it logged nothing precisely when commands failed).

### Distinct from related issues
- Not #253 (panel auto-refresh) -- the file is genuinely *not written* here, not "written but not displayed."
- Distinct from the display-refresh angle reported separately for gpt-5.5.

### Fix direction (brain-guidance)
Strengthen notebook-logging steering for weaker models -- e.g. an end-of-turn nudge or deterministic reminder to log attempts/outcomes to notebook.md, especially after failed commands. Fits #99 (right-size prompt for smaller models). A deterministic notebook-append hook for key session events would make logging less dependent on model compliance.

### Environment
Orbit 0.4.1, Linux x64; failing model openai/gpt-4o, working model google/gemini-flash-latest.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.