Custom endpoints can send unsupported sampling parameters
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
- Copilot Chat Extension Version: `0.66.0`
- VS Code Version: `1.138.0` (Windows x64)
- OS Version: Windows
- Feature (e.g. agent/edit/ask mode): Agent mode
- Selected model: A model configured through `customendpoint` (kimi-k3 from moonshot)
- Logs:
```text
HTTP 400 invalid_request_error: invalid sampling parameter for this model
The selected model requires a fixed sampling configuration, but the request
contains an incompatible temperature or top_p value.
The error is reproducible in Agent mode. Changing the global Agent sampling
setting to match the model's required value avoids the error.
```
The full Copilot Chat log can be provided privately if needed. The attached
public description intentionally omits model identifiers, endpoint URLs,
request bodies, and credentials.
Steps to Reproduce:
# Bug: custom endpoints can send unsupported sampling parameters
Some models exposed through `customendpoint` require specific sampling parameters. For example, a model may only accept a fixed `temperature` value or a fixed nucleus-sampling value. However, when such a model is used in Chat/Agent mode, VS Code can send generic agent sampling options instead of the values required by the model.
This produces requests such as:
```json
{
"model": "model-specific-id",
"temperature": 0,
"top_p": 1
}
```
The upstream model provider rejects the request because the sampling parameters are invalid for that model.
```text
400 invalid sampling parameter for this model
```
## Reproduction
Configure a model through `customendpoint`, including model-specific sampling options:
```json
{
"vendor": "customendpoint",
"apiType": "chat-completions",
"models": [
{
"id": "model-specific-id",
"name": "Model",
"toolCalling": true,
"vision": true,
"maxInputTokens": 840000,
"maxOutputTokens": 128000,
"modelOptions": {
"temperature": 1,
"top_p": 0.95
}
}
]
}
```
Then send a request from the Chat Agent while the generic agent sampling options differ from the model's required values. The request fails unless the user manually changes a global agent setting, even though the model configuration already declares the required values.
## Root cause
The current BYOK request path has two relevant behaviors:
1. Model-specific defaults can be declared in the custom endpoint configuration.
2. The BYOK endpoint subsequently applies per-request sampling options. Agent requests provide sampling values through `requestOptions`, and those values take precedence over the model's configured `modelOptions`.
As a result, a model-level configuration such as `modelOptions.temperature` is not sufficient for models whose APIs require fixed or restricted sampling values. Users currently need to know about and change an unrelated global agent setting.
## Expected behavior
Model-specific sampling constraints should apply at the final `customendpoint` request-body boundary, after per-request and model-level options have been merged.
Conceptually:
```ts
const body = createCapiRequestBody(options, this.model, this.getCompletionsCallback());
this._applyReasoningEffort(body, options);
this._applyConfiguredModelOptions(body, options);
// Apply the selected model's required sampling constraints here.
applyModelSamplingConstraints(body, this.modelMetadata);
return body;
```
The same invariant should be preserved for the applicable request formats supported by the endpoint. A model's declared sampling constraints should take precedence over generic agent sampling defaults.
This keeps ordinary models unchanged and avoids requiring users to configure global agent settings for a single model.
## Suggested implementation direction
The final request body should be normalized using the selected model's declared constraints. The relevant tests should verify that:
- A model with fixed sampling constraints emits its required `temperature` and `top_p` values when the request supplies different values.
- The behavior covers constraints declared through model metadata or configuration.
- Models without sampling constraints continue to honor explicit per-request sampling values.
- `modelOptions` values continue to work for ordinary models.
The existing custom endpoint tests should be extended rather than introducing a separate test abstraction where possible.
## Validation details
I reproduced this with VS Code's `customendpoint` configuration. Before the fix, Agent mode sent sampling values that were invalid for the selected model. Changing a global agent sampling setting avoids the error, which confirms that the failure is caused by request-option precedence rather than authentication or endpoint selection.
The proposed change should be validated with the focused BYOK endpoint test suite and a manual request using:
- a model with fixed sampling constraints through `customendpoint`
- Agent mode with the default sampling values
- a model without fixed constraints using the same endpoint type
## Related documentation
- [VS Code contribution guide](https://github.com/microsoft/vscode/wiki/How-to-Contribute)
- [BYOK OpenAI endpoint implementation](https://github.com/microsoft/vscode/blob/main/extensions/copilot/src/extension/byok/node/openAIEndpoint.ts)
- [Model capability handling](https://github.com/microsoft/vscode/blob/main/extensions/copilot/src/platform/endpoint/node/chatEndpoint.ts)
Contributor guide
Assessment
This issue has not been assessed yet.