Custom "anthropic" provider does not enforce provider.max_prompt_tokens — sessions grow past the model context window until a hard 400
- 主要语言
- Java
- 星标
- 10.5k
- 派生
- 1.5k
- 平均合并
- 1 天 11 小时
- 30 天内合并 PR
- 127
描述
## Summary
When a custom provider of `type: "anthropic"` is configured with `max_prompt_tokens`, the SDK does not appear to enforce that budget. A long-running session keeps accumulating transcript until the request exceeds the model's context window and the provider rejects it with HTTP 400 -- after which the session is permanently unusable.
## Configuration
The provider is created on every session create/resume with an explicit prompt budget:
```json
{
"type": "anthropic",
"base_url": "...",
"model_id": "claude-sonnet-5",
"max_output_tokens": 32768,
"max_prompt_tokens": 967232
}
```
`max_prompt_tokens` is derived as `context_window (1,000,000) - max_output_tokens (32,768) = 967,232`.
## Expected
The SDK compacts (or otherwise bounds the prompt) before crossing `max_prompt_tokens = 967232`.
## Actual
The transcript grew unbounded to **1,001,142 tokens** -- 33,910 past the configured budget, and past the model's 1,000,000 hard limit:
```
400 invalid_request_error
"prompt is too long: 1001142 tokens > 1000000 maximum"
```
The session had run ~18 successful turns over ~3 hours, with the serialized request growing steadily (~1.98 MB -> ~2.04 MB) before crossing the limit. Tool count was constant throughout, so the growth is accumulated conversation history rather than tool schemas.
## Two additional observations
1. **`infinite_sessions` thresholds also appear inert on this path.** `background_compaction_threshold` / `buffer_exhaustion_threshold` are sent on every turn but appear to have no effect for the `anthropic` provider (they do take effect on the Copilot backend path). So neither the threshold-based compaction nor the `max_prompt_tokens` budget bounded the transcript.
2. **The session actively degrades after the first failure.** Once over the limit, continued turns keep *appending* to the transcript -- request size grew from ~2.044 MB to ~2.065 MB across ~30 consecutive failed turns. There is no back-off, trim, or compaction triggered by the 400, so the session can never self-recover; every subsequent turn fails immediately (~1.5s vs. the 38s first failure).
## Impact
Every turn in an affected session fails permanently. The only recovery is to start a new session, and nothing in the surfaced error indicates that to the user. Because the failure is a deterministic 400, retry suppression correctly kicks in -- but that just means the session is durably wedged.
## Environment
- SDK 1.0.7 / Copilot CLI 1.0.71
- Custom `anthropic` provider over an OpenAI-incompatible relay endpoint
- Model `claude-sonnet-5` (1,000,000-token context window)
## Ask
Should `provider.max_prompt_tokens` be enforced on the `anthropic` provider path (and/or should `infinite_sessions` compaction apply there)? If enforcement is intentionally backend-only today, it would help to document that clearly, since the field is accepted without warning and silently has no effect.
贡献指南
调研方向
Start at the custom anthropic provider session-create/resume path and trace how max_prompt_tokens and infinite_sessions thresholds are handled before requests are sent. Compare that path with the Copilot backend path where compaction works, then reproduce the growing transcript and confirm that prompts stay within the configured budget and failed sessions can recover.
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- api, backend
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 活跃
- 描述清晰度
- 基本清楚
- 新手友好度
- 48/100