speakeasy-api / speakeasy-api/kit

Model downswitch can exceed the target context window

Open
#18 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
40
Forks
4
Avg merge
4h 18m
Merged PRs (30d)
148

Description

Kit can switch the main ACP session to a model with a smaller context window without first checking whether the existing transcript fits that model. The next turn keeps the live transcript and creates the selected provider session, so the target provider can reject the request for exceeding its context limit.

Automatic compaction does not currently prevent this. Its trigger uses the latest provider-reported usage and context-window metadata in the transcript, which still describes the previously active model. After the switch, manual or automatic compaction also follows the new model and may fail because its summarization request contains the already oversized transcript.

Until this is fixed, compact with the larger model before switching:

/compact
/model <lower-context-model>

A safe downswitch should be transactional and serialized through the session actor. Kit should retain context-window information in the model catalog, compare the transcript against the target window, compact with the currently active model before publishing the new selection, persist the replacement transcript, and then apply the switch. If the target window is unknown or compaction cannot make the transcript fit, Kit should leave the current selection unchanged and return an actionable error.

Relevant implementation: src/provider/adapter.rs (SelectableSession::begin_turn), src/protocols/acp.rs (Command::SetModel and set_model), and src/compaction.rs (automatic, compaction_reason, and KitCompactionBackend).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with SelectableSession::begin_turn in src/provider/adapter.rs, then trace Command::SetModel and set_model in src/protocols/acp.rs and the automatic, compaction_reason, and KitCompactionBackend paths in src/compaction.rs. Check how the session actor serializes selection and how transcript metadata is stored. Done means an unsafe downswitch compacts and persists first, while unknown or oversized targets leave the current model unchanged with an actionable error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai-infra-agents, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.