microsoft / microsoft/vscode

Feature: Support extraBody configuration for BYOK custom models

Open
#336,171 1 comment 1 reaction 1 assignee Claimed by @vritant24 View on GitHub
feature-request model-byok
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

## Summary

Allow BYOK (Bring Your Own Key) custom models to configure additional request body fields via an `extraBody` property in `chatLanguageModels.json`. This would enable users to pass model-specific parameters that are not currently supported through the existing `modelOptions` configuration.
(**This text was created with Copilot!**)

## Use Case

When using custom OpenAI-compatible endpoints (like vLLM, local inference servers, or specialized APIs), models often require additional parameters beyond the currently supported `temperature` and `top_p`:

- `reasoning_effort` (for reasoning models)
- `preserve_thinking` (to retain internal reasoning in responses)
- `chat_template_kwargs` (for custom chat template configuration)
- Provider-specific parameters for cache control, routing, or session management
- Custom model options (e.g., `output_config`, nested configuration objects)

### Example Configuration

```json
{
"vendor": "customendpoint",
"models": [
{
"id": "GLM-5.3-Flash-EXL3",
"name": "2xDGX-GLM-5.3-Flash-EXL3",
"url": "http://myAIServer:8600/v1",
"toolCalling": true,
"vision": true,
"maxInputTokens": 262144,
"maxOutputTokens": 32768,
"modelOptions": {
"temperature": 0.2,
"top_p": 0.95
},
"extraBody": {
"reasoning_effort": "none",
"preserve_thinking": false,
"chat_template_kwargs": {
"enable_thinking": false
}
}
}
]
}
```

## Current Limitations

1. **Only `temperature` and `top_p` are supported** in `modelOptions` (see `byokProvider.ts` lines 308-321 and `openAIEndpoint.ts` lines 302-324)
2. **No generic body customization** - users cannot pass arbitrary request body fields
3. **Model initialization failures** - models that require additional parameters cannot be properly configured
4. **Inconsistent with other extensions** - some extensions (e.g., Kilo Code) already support custom header and body configuration

## Proposed Solution

### Option A: Flexible `extraBody` Field (Recommended)

Add an optional `extraBody` field to model capabilities that allows arbitrary JSON to be merged into the request body:

```typescript
export interface BYOKModelCapabilities {
// ... existing fields ...
/**
* Additional fields to merge into the request body.
* These override default values but can be overridden by per-request parameters.
*/
extraBody?: Record;
}
```

**Implementation:**
- Merge `extraBody` into the request body after model options are applied
- Deep merge for nested objects
- Per-request parameters take precedence

### Option B: Explicit Parameter List

Extend `modelOptions` to support common additional parameters:
- `reasoning_effort`
- `preserve_thinking`
- Custom output configuration options

**Pros:** More type-safe, better validation
**Cons:** Less flexible, requires updating schema for each new parameter

## Additional Context

- Related to #316794 (custom headers request)
- Affects users of local deployments (vLLM, ollama, custom inference servers)
- Impacts compatibility with specialized reasoning models (DeepSeek, Claude with thinking)
- See #312746 for related issue with reasoning content handling

## Suggested Implementation Path

1. Add `extraBody?: Record` to `BYOKModelCapabilities` interface
2. Update JSON schema in `package.json` to allow `extraBody` object
3. Modify `OpenAIEndpoint._applyConfiguredModelOptions()` to merge `extraBody` into request body
4. Ensure per-request parameters still take precedence
5. Add unit tests for body merging behavior
6. Document in language model customization guide

## Related Issues

- #316794 - Support custom headers or stable chat session identifiers for BYOK
- #312746 - Copilot Chat drops reasoning_content field with DeepSeek models
- #319968 - Optional model discovery for customendpoint
- #318748 - Custom endpoint model display issues

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.