anomalyco / anomalyco/opencode
[FEATURE]: Auto-switch to vision model when images enter the agent loop (incl. tool-produced)
@nexxeln is already working on this.
Since Sep 2, 2026.
- Dominant language
- TypeScript
- Stars
- 209k
- Forks
- 27.5k
- PR merge metrics
- PR metrics pending
Description
- I have searched for existing issues and confirmed this is not a duplicate
- I have read the contributing guidelines
Problem
Many teams pair a strong text-only coding model (which cannot read images) with a companion vision model. Currently there is no automatic way to use the vision model exactly when an image needs to be understood, and the text model otherwise.
The painful case is autonomous work inside the agent loop: tools such as Playwright (screenshots), Figma, or the filesystem read tool return image attachments mid-task. The active model stays text-only for the rest of that turn and simply cannot see the image — forcing the user to manually swap the model, which (per #46345) then leaves the images unusable anyway.
Expected behavior
Automatically switch the active model to a configured vision model whenever an image enters the conversation context, and switch back to the primary text model when no images remain:
- Image attached in a new user message → switch to vision before the first provider call.
- Image produced by a tool (Playwright screenshot, Figma export, filesystem read of an image) mid-loop → switch to vision before the next provider turn.
- When no images are present in the current turn → switch back to the text model.
The key technical gap (why existing issues don't cover this)
In the classic agent loop (packages/opencode/src/session/prompt.ts, runLoop), the model is pinned to the last user message and re-resolved on every iteration:
const model = yield* getModel(lastUser.model.providerID, lastUser.model.modelID, sessionID)
switchModel() / POST /api/session/{sessionID}/model only updates session.model. Because lastUser.model stays unchanged, the in-flight loop keeps the original text-only model for the remaining turns of that user message. So a screenshot produced by a tool in turn N is still consumed by the text-only model in turn N+1, regardless of any model switch.
The newer runner (core/session/runner/llm.ts) already resolves the model fresh per turn via models.resolve(session) — which would make an automatic switch apply immediately. This feature would benefit from making model resolution per provider turn in the stable path, matching that behavior.
Proposed API
A small config option, mirroring the shape requested in #31936:
{
"model": "<text-only-model>", // primary text model
"vision_model": "<vision-model>" // used when images are in context
}
Optionally also expose a plugin hook (e.g. a tool.result / turn event carrying whether the context now contains an image) so plugins can drive the switch programmatically and detect the specific "image to be sent next turn" condition.
Detection
Reuse the existing modalities/capabilities.input.image field on models to decide whether the target model supports vision, and match image FileParts by mime starting with image/ (including ToolPart attachments).
Related issues (for maintainer triage — not duplicates)
- #46345 — Bug: switching text-only → vision leaves images unusable. This is the failure the auto-switch must avoid; fixing #46345 is complementary.
- #33199 / PR #33471 — Fixed UI-driven model switch server sync, but not mid-loop continuation.
- #31936, #22828, #24948 — Propose a "vision sub-model transcription fallback" (image → text, continue on main model). This request differs: it switches the model/persona of the whole session to the vision model, rather than transcribing into text.
- #34370 — Broader multi-model orchestration (intent routing, advisor). This is the focused slice: automatic vision switching for image-bearing turns.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.