Feature request: Capability-based model routing (route to vision model only when image input is present)

Summary

Request a capability-based / conditional model routing mechanism: route a task to a vision-capable model only when the task input contains images, and to a text-only model otherwise. This is more granular than the static per-role model assignment proposed in #683, and it directly addresses the production failure mode described in #755.

Problem

We run an autonomous multi-agent CI/CD pipeline on jcode with 4 specialized roles (issue triage, implementation, code review, end-to-end verification). Three of the four roles are purely text-based (code diffs, curl+grep on preview URLs, log analysis) and run best on a cheap text-only model (glm-5.2). The fourth role (triage) needs to read image attachments (mockups, screenshots) that users attach to issues — but only sometimes: many issues are text-only and have no images.

Today we have two bad options:

1. Run all roles on a vision model — 3x the cost for 3 roles that never touch an image.
2. Run all roles on a text-only model — triage crashes (#755) or silently fabricates image content when an issue has image attachments.

The per-role model override proposed in #683 ([swarm.roles.<role>].model) gets us partway, but it is static: triage would always use the vision model, even on text-only issues where the text-only model is cheaper and sufficient. What we want is dynamic routing based on task input:

"If this task's input contains image content (attachment, screenshot, browser capture), route to a vision-capable model. Otherwise, route to the text-only model."

Proposed mechanism

A routing policy that inspects the task input (or the node's declared capability requirements) and selects the model accordingly. Possible shapes:

Option A — Input-content-based routing (automatic)

The runtime inspects the message history / task input for ContentBlock::Image (or declared image attachments) and routes to a vision-capable model when present, falling back to the text-only model otherwise. Config:

[routing.vision]
on_image_input = "provider/vision-model"
fallback = "provider/text-only-model"

This pairs naturally with the #755 fix (auto-strip images for text-only models): if a vision model is available and routed to, images are kept; if not, they're stripped and the text-only model handles the text portion.

Option B — Per-node capability declaration (DAG-aware)

In the swarm DAG model (SWARM_TASK_GRAPH.md), each node declares its capability requirements, and the scheduler picks a worker from the pool that satisfies them:

[swarm.nodes.triage_with_images]
requires = ["vision"]
model = "provider/vision-model"

[swarm.nodes.triage_text_only]
requires = []
model = "provider/text-only-model"

The DAG expansion step would branch on "does this issue have image attachments?" and emit the appropriate node. This aligns with the "per-node prompts" direction already hinted in SWARM_TASK_GRAPH.md.

Option C — Provider-pool with capability tags

The provider pool is tagged with capabilities (vision, text-only, tool-use, etc.) and the router picks any provider that satisfies the task's declared requirements:

[[providers]]
name = "glm-text"
model = "glm-5.2"
capabilities = ["text-only"]

[[providers]]
name = "vision-model"
model = "..."
capabilities = ["vision", "text"]

Why this is distinct from #683

#683 proposes static per-role model assignment ("coder → fast model, reviewer → strong model"). That's a role-level decision made once at config time.

This request is for dynamic per-task routing within a single role, based on the task's actual input. The same role (triage) may need a vision model on one issue and a text-only model on the next. Static per-role routing can't express that — it would force the vision model on every triage run, or the text-only model on every triage run, but not "vision when images, text-only otherwise."

Both are valuable; they compose. #683 gives per-role defaults, this gives per-task overrides based on input capabilities.

Use case (concrete)

Our triage role processes incoming issues. An issue may contain:

• Text only → text-only model is sufficient and cheaper.
• Text + image attachments (mockup, screenshot, logo) → vision model required to actually see the image instead of fabricating a description.

With capability-based routing, triage runs on glm-5.2 (text-only, cheap) for 80% of issues and auto-escalates to a vision model for the 20% that have images — without us writing per-issue branching logic in the agent prompt. The runtime handles the routing; the agent prompt stays model-agnostic.

Related

• #683 — per-agent/per-role model selection (static). This issue is the dynamic counterpart.
• #755 — image blocks break text-only sessions. Capability-based routing would prevent images from ever reaching a text-only model in the first place, making #755's auto-strip a defense-in-depth rather than the primary fix.

Start by reading SWARM_TASK_GRAPH.md and the related issues #683 and #755 to understand the existing model-selection and image-handling directions. Determine which routing shape is intended before locating the runtime and configuration entry points. Done means image-bearing tasks reach a vision-capable model while text-only tasks use the fallback, with coverage for both paths.