galaxyproject / galaxyproject/loom
Agent reasoning/preamble leaks into the visible chat as a text block (observed on Gemini 3.5 Flash)
- Dominant language
- TypeScript
- Stars
- 14
- Forks
- 12
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 17
Description
## Symptom
The agent's internal planning/reasoning is rendered to the user as part of the chat answer. For a trivial prompt, the visible response began with the model talking to itself:
> **Prompt:** `ok. create a new section in the notebook called "Galaxy implementation"`
>
> **Visible response (verbatim start):** *"Excellent! All 77/77 checks pass successfully (100% PASS!). We are completely done with this turn. Let's summarize the progress and let the user know what was done. I will write a polished, direct response. I won't narrate step-by-step in chat and keep it professional and direct. I will outline that the 'Galaxy implementation' section has been created…"* — and only then the actual answer.
That whole preamble (including the model **reciting its own formatting instructions** — "I won't narrate step-by-step in chat") should never reach the user.
Observed while running on **Gemini 3.5 Flash** (`google` / `gemini-3.5-flash`).
## Evidence (run log)
The leaked text arrived as a single **text** content block, not a thinking block:
```
[orbit] event: message_update {assistantMessageEvent:{type:"text_delta","contentIndex":0,
"delta":"! All 77/77 checks pass successfully (100% PASS"...}}
[orbit] event: message_update {...text_delta,"contentIndex":0,"delta":".\nI will write a polished, direct response. I won't n..."}}
[agent] ← event: message_end {message:{role:"assistant","content":[{"type":"text","text":"Excellent! All 77/77 checks pass...
```
The entire monologue is `content[0]` `type:"text"` — i.e. the model's reasoning/preamble was surfaced in the **visible output-text channel** rather than kept in a `thinking` block (other turns in the same session *did* emit proper `thinking_start`/`thinking_end` blocks, so it's inconsistent).
## Likely cause
The model's reasoning/preamble is landing in the output-text stream instead of the thinking channel. This is provider/model behavior in how reasoning vs. final output is separated — Orbit faithfully renders whatever it's handed as `text`, so the planning leaks. Fix belongs in the provider adapter / model handling (pi-ai), not the renderer: route reasoning to the thinking block, and/or detect-and-suppress a leading planning preamble before the final answer.
## Acceptance criteria
- [ ] Internal planning/reasoning (incl. "I will write a polished response / I won't narrate…" style monologue) does not appear in the visible chat answer.
- [ ] Reasoning is shown only in the thinking/collapsible channel, consistently across turns and providers.
- [ ] Verified on Gemini 3.5 Flash (and spot-checked on other reasoning-capable models).
## Related
- #206 — chat output running together between streamed segments (adjacent rendering/streaming area).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the provider adapter/model handling path for Gemini 3.5 Flash and trace how message_update text and thinking blocks are mapped into visible chat output. Reproduce the reported run, then verify that planning text is consistently kept in the thinking channel and absent from the final answer across Gemini and another reasoning-capable model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100