feature: transparent model and provider switching — continue a session across a limit, price or policy boundary
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 13
- Forks
- 40
- Avg merge
- 12h 17m
- Merged PRs (30d)
- 156
Description
Feature Description
@rubambiza's idea: when a session approaches a token or rate limit, Cortex switches the request
to a different model or provider in the background, without interrupting the user's work —
the agent keeps going instead of stalling on a 429 or a context-window error.
The same mechanism serves three other triggers:
- Cost — route to an equivalent-quality, cheaper model or provider when one is available
- Preference — a user or team pins a preferred model per task class
- Policy / compliance — a request that would violate a data-residency, provider-approval or
region constraint is routed to a compliant provider, or refused with a clear reason rather
than silently sent
Cortex is uniquely placed to do this: it already sits in the request path with the request body
decrypted, and already rewrites bodies (Context Guru compaction, tool-definition pruning). This
is a new plugin in that pipeline, not new architecture.
Proposed Solution
- Provider abstraction: translate a request between provider wire formats (Anthropic
Messages ↔ OpenAI Chat Completions ↔ others) including tool definitions, tool results,
system prompts and streaming events - Trigger evaluation before the upstream call: projected token count vs the target model's
window, observed rate-limit headroom, price table, policy rules - Retry-on-limit path: a 429 or context-overflow from the primary is re-issued against the
fallback rather than surfaced to the agent - Preserve streaming — the client must not be able to tell a switch happened mid-turn, and
the fix for #926 (early flush of leading thinking block) applies here too - Credentials for each provider resolved through AuthBridge, so the agent never holds more
than one provider's secret - Every switch is recorded and visible. A silent model change is a correctness hazard:
the user must be able to see, per request, which model actually answered, why it was
chosen, and what it cost. Depends on #950 - Opt-in, off by default, with a documented list of substitutions the user has approved
Risks and open questions
- Silent quality change is the real risk. "Without interrupting your work" and "you should
know which model answered you" are in tension. Proposal: never interrupt, always record, and
surface a visible marker inabctland in the session summary - Semantics are not preserved across providers: prompt caching, thinking blocks, tool-call
formats and stop reasons all differ. A switch mid-session can invalidate a cache the user was
paying to build — the cost model has to account for that - Whether policy enforcement belongs here or in the existing egress-control / guardrails plugins
- Does the agent's own context (conversation history) transfer cleanly, or does a switch
effectively start a new upstream cache lineage?
Additional Context
Credit: @rubambiza — team brainstorming, September 2026. Post-v0.9.0 — this is the largest of the
four brainstorm items and would not fit the release window.
Foundational for #1020 (task-class routing) and #1021 (parallel dual-model comparison), which
both need the provider abstraction and the switch mechanism this issue builds.
Depends on #950 for per-model cost attribution.
Part of the post-v0.9.0 brainstorm backlog under #962.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by mapping the request path and existing Context Guru, egress-control or guardrails, and AuthBridge plugins, then read the constraints in #950 and the streaming concern in #926. Done requires a scoped provider abstraction and switch design that resolves the policy, cache, streaming, attribution, and visibility questions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend-api-design, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100