anomalyco / anomalyco/opencode

Feature: Dynamic context length detection for OpenAI-compatible providers

Open
#40,908 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
209k
Forks
27.5k
PR merge metrics
PR metrics pending

Description

Summary

When using a custom OpenAI-compatible provider (via @ai-sdk/openai-compatible), opencode does not dynamically detect the context length of the currently loaded model from the server's /models endpoint. Instead, it requires limit.context to be hardcoded in the config for each model.

Problem

Local model servers (LM Studio, Unsloth Studio, etc.) expose context_length for the loaded model in their /models API response. However, opencode ignores this and requires static configuration. This means:

  1. Users must manually set limit.context for every model
  2. If the user changes the context size on the server, opencode doesn't know
  3. Different models may have different context lengths, requiring constant config updates

Expected Behavior

opencode should query the provider's /models endpoint at runtime and read the context_length field to determine the appropriate context window for the currently loaded model.

Example Server Response

{
  "object": "list",
  "data": [
    {
      "id": "DavidAU/Qwen3.5-9B-...",
      "object": "model",
      "owned_by": "unsloth-studio",
      "context_length": 262144,
      "max_context_length": 208128,
      "native_context_length": 262144,
      "loaded": true
    }
  ]
}

Current Workaround

Users must manually specify limit.context (and limit.output) in opencode.json for each model definition under provider.<name>.models.

Suggested Solution

For OpenAI-compatible providers, fetch the /models endpoint at connection time and use the context_length from the loaded model's metadata as the context window. This could be refreshed periodically or on model switch.

Environment

  • Provider type: @ai-sdk/openai-compatible
  • Local server: Unsloth Studio (also applies to LM Studio, Ollama, etc.)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the @ai-sdk/openai-compatible provider integration and how provider..models configuration supplies limit.context. Then inspect how the OpenAI-compatible /models response is accessed or could be queried at connection time. Done means the loaded model's context_length is used dynamically, including after a model switch or server-side context-size change, without requiring a hardcoded value.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.