Flexible Context Window & Custom Thinking Effort for BYOK
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
*Additional part of #325237*
Add these 2 new features for Custom Endpoint:
1. Add flexible context window.
Add `availableContextWindow`:
```jsonc
{
"id": "custom",
// ...
"maxInputTokens": 1024000,
"maxOutputTokens": 64000, // `maxTokens` - `maxInputTokens` or 16k if not defined
"maxTokens": 1024000,
"availableContextWindow": [200000, 500000, 1000000] // Use the smallest CWS by default to reduce token consumption.
}
```
> [!NOTE]
> When both `availableContextWindow` and `maxInputTokens`/`maxTokens` are defined, use the former first.
2. Add custom thinking effort.
Add `effort`:
```jsonc
{
"id": "custom",
// ...
"efforts": ["off", "low", "medium", "high", "xhigh", "max", "minimal"]
// Display name: Off, Low, Medium, High, **Extra High**, Max, Minimal (usually for Gemini). Other efforts will be shown as-is.
"defaultEffort": "medium" // Default: Closest to `medium`. It cannot be `max`.
"useLegacyThinking": false // For older models which do not support efforts. Conflicts with `efforts` and `defaultEffort`.
}
```
3. Automatically save user's configuration of the settings above, except Max Effort.
Contributor guide
Assessment
This issue has not been assessed yet.