1jehuang / 1jehuang/jcode

Feature: configurable effective context-window ceiling (below model max) that auto-compact targets

Open
#723 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

autonomous: no enhancement priority: medium triage: needs-decision
Dominant language
Rust
Stars
19.9k
Forks
2.3k
Avg merge
2d 7h
Merged PRs (30d)
30

Description

Summary

Feature request: a configurable effective context-window ceiling that can be set lower than a model's native max context, with auto-compact triggering relative to that ceiling — not the model's raw context_window().

Current behavior

  • Provider::context_window() resolves to the model's native max (e.g. 1,000,000 for claude-opus-5, 200,000 for claude-sonnet-4-5) — see crates/jcode-provider-anthropic-runtime/src/context_window.rs.
  • jcode-compaction-core::COMPACTION_THRESHOLD (0.80) and CRITICAL_THRESHOLD (0.95) are hardcoded fractions applied directly to that native max (crates/jcode-base/src/compaction.rs).
  • The only existing user override, context_window (alias context_limit) in [[providers.<name>.models]], is documented as filling in a value when the provider has no usable /v1/models catalog response — it's meant to report the true max, not to cap it lower for operational reasons.

There's currently no way to say "this model supports 1M tokens, but I want jcode to always compact once we hit e.g. 100K, regardless."

Why this matters

On models with very large native windows (e.g. Opus 5 at 1M), a session can run for a very long time and accumulate a huge amount of context before the 80%/95% thresholds ever fire. In practice this means:

  • Much higher per-turn cost and latency as the transcript grows, well before compaction ever kicks in.
  • Degraded model attention/quality on long, sprawling contexts even when technically "within budget."
  • Less frequent, much larger compaction events instead of smaller, more frequent ones — worse for keeping a tight working set.

Being able to set a lower operational ceiling (e.g. "treat this model as if it only has 120K tokens of context, even though it can do 1M") lets a user trade off cost/quality/latency deliberately, independent of what the model can technically hold.

Suggested fix

  • Add a config option (e.g. effective_context_window or context_window_cap, global and/or per-provider/per-model) that, when set, is used as the base for COMPACTION_THRESHOLD/CRITICAL_THRESHOLD math instead of provider.context_window().
  • Should be clearly distinguished from the existing context_window/context_limit field (which reports the model's true capacity for catalog-less providers) — this is a policy cap, not a capability report.
  • The TUI context-usage meter should reflect usage against the capped ceiling when set, so the displayed percentage matches when auto-compact will actually fire.
  • Reasonable to validate that the cap is <= the model's actual resolved context window (silently clamp or warn, not silently ignored).

Related code pointers

  • crates/jcode-compaction-core/src/lib.rs:9,13COMPACTION_THRESHOLD/CRITICAL_THRESHOLD constants.
  • crates/jcode-base/src/compaction.rs — threshold application (try_auto_compact_after_context_limit, hard/soft threshold wait logic).
  • crates/jcode-provider-anthropic-runtime/src/context_window.rs and equivalents in other provider-runtime crates — context_window() resolution.
  • crates/jcode-config-types/src/lib.rs:353CompactionConfig struct, natural home for a new cap field.
  • crates/jcode-config-types/src/lib.rs:422NamedProviderModelConfig — existing context_window/context_limit field for comparison/contrast.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CompactionConfig in crates/jcode-config-types/src/lib.rs:353 and compare NamedProviderModelConfig at line 422, then trace threshold application in crates/jcode-base/src/compaction.rs. Review context_window() resolution in the provider-runtime crates and the TUI context-usage meter. Done means a validated operational ceiling can be configured, auto-compaction uses it, and displayed usage reflects the same ceiling.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai, cli, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.