rossoctl / rossoctl/cortex

feature: transparent model and provider switching — continue a session across a limit, price or policy boundary

Open
#1,019 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
13
Forks
40
Avg merge
12h 17m
Merged PRs (30d)
156

Description

Feature Description

@rubambiza's idea: when a session approaches a token or rate limit, Cortex switches the request
to a different model or provider in the background, without interrupting the user's work
the agent keeps going instead of stalling on a 429 or a context-window error.

The same mechanism serves three other triggers:

  • Cost — route to an equivalent-quality, cheaper model or provider when one is available
  • Preference — a user or team pins a preferred model per task class
  • Policy / compliance — a request that would violate a data-residency, provider-approval or
    region constraint is routed to a compliant provider, or refused with a clear reason rather
    than silently sent

Cortex is uniquely placed to do this: it already sits in the request path with the request body
decrypted, and already rewrites bodies (Context Guru compaction, tool-definition pruning). This
is a new plugin in that pipeline, not new architecture.

Proposed Solution
  • Provider abstraction: translate a request between provider wire formats (Anthropic
    Messages ↔ OpenAI Chat Completions ↔ others) including tool definitions, tool results,
    system prompts and streaming events
  • Trigger evaluation before the upstream call: projected token count vs the target model's
    window, observed rate-limit headroom, price table, policy rules
  • Retry-on-limit path: a 429 or context-overflow from the primary is re-issued against the
    fallback rather than surfaced to the agent
  • Preserve streaming — the client must not be able to tell a switch happened mid-turn, and
    the fix for #926 (early flush of leading thinking block) applies here too
  • Credentials for each provider resolved through AuthBridge, so the agent never holds more
    than one provider's secret
  • Every switch is recorded and visible. A silent model change is a correctness hazard:
    the user must be able to see, per request, which model actually answered, why it was
    chosen, and what it cost. Depends on #950
  • Opt-in, off by default, with a documented list of substitutions the user has approved
Risks and open questions
  • Silent quality change is the real risk. "Without interrupting your work" and "you should
    know which model answered you" are in tension. Proposal: never interrupt, always record, and
    surface a visible marker in abctl and in the session summary
  • Semantics are not preserved across providers: prompt caching, thinking blocks, tool-call
    formats and stop reasons all differ. A switch mid-session can invalidate a cache the user was
    paying to build — the cost model has to account for that
  • Whether policy enforcement belongs here or in the existing egress-control / guardrails plugins
  • Does the agent's own context (conversation history) transfer cleanly, or does a switch
    effectively start a new upstream cache lineage?
Additional Context

Credit: @rubambiza — team brainstorming, September 2026. Post-v0.9.0 — this is the largest of the
four brainstorm items and would not fit the release window.

Foundational for #1020 (task-class routing) and #1021 (parallel dual-model comparison), which
both need the provider abstraction and the switch mechanism this issue builds.
Depends on #950 for per-model cost attribution.

Part of the post-v0.9.0 brainstorm backlog under #962.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by mapping the request path and existing Context Guru, egress-control or guardrails, and AuthBridge plugins, then read the constraints in #950 and the streaming concern in #926. Done requires a scoped provider abstraction and switch design that resolves the policy, cache, streaming, attribution, and visibility questions.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend-api-design, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.