microsoft / microsoft/vscode

Custom endpoints can send unsupported sampling parameters

Open
#336,736 0 comments 0 reactions 1 assignee Assigned to @vritant24 View on GitHub
new release
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

- Copilot Chat Extension Version: `0.66.0`
- VS Code Version: `1.138.0` (Windows x64)
- OS Version: Windows
- Feature (e.g. agent/edit/ask mode): Agent mode
- Selected model: A model configured through `customendpoint` (kimi-k3 from moonshot)
- Logs:

```text
HTTP 400 invalid_request_error: invalid sampling parameter for this model

The selected model requires a fixed sampling configuration, but the request
contains an incompatible temperature or top_p value.

The error is reproducible in Agent mode. Changing the global Agent sampling
setting to match the model's required value avoids the error.
```

The full Copilot Chat log can be provided privately if needed. The attached
public description intentionally omits model identifiers, endpoint URLs,
request bodies, and credentials.

Steps to Reproduce:

# Bug: custom endpoints can send unsupported sampling parameters

Some models exposed through `customendpoint` require specific sampling parameters. For example, a model may only accept a fixed `temperature` value or a fixed nucleus-sampling value. However, when such a model is used in Chat/Agent mode, VS Code can send generic agent sampling options instead of the values required by the model.

This produces requests such as:

```json
{
"model": "model-specific-id",
"temperature": 0,
"top_p": 1
}
```

The upstream model provider rejects the request because the sampling parameters are invalid for that model.

```text
400 invalid sampling parameter for this model
```

## Reproduction

Configure a model through `customendpoint`, including model-specific sampling options:

```json
{
"vendor": "customendpoint",
"apiType": "chat-completions",
"models": [
{
"id": "model-specific-id",
"name": "Model",
"toolCalling": true,
"vision": true,
"maxInputTokens": 840000,
"maxOutputTokens": 128000,
"modelOptions": {
"temperature": 1,
"top_p": 0.95
}
}
]
}
```

Then send a request from the Chat Agent while the generic agent sampling options differ from the model's required values. The request fails unless the user manually changes a global agent setting, even though the model configuration already declares the required values.

## Root cause

The current BYOK request path has two relevant behaviors:

1. Model-specific defaults can be declared in the custom endpoint configuration.
2. The BYOK endpoint subsequently applies per-request sampling options. Agent requests provide sampling values through `requestOptions`, and those values take precedence over the model's configured `modelOptions`.

As a result, a model-level configuration such as `modelOptions.temperature` is not sufficient for models whose APIs require fixed or restricted sampling values. Users currently need to know about and change an unrelated global agent setting.

## Expected behavior

Model-specific sampling constraints should apply at the final `customendpoint` request-body boundary, after per-request and model-level options have been merged.

Conceptually:

```ts
const body = createCapiRequestBody(options, this.model, this.getCompletionsCallback());

this._applyReasoningEffort(body, options);
this._applyConfiguredModelOptions(body, options);

// Apply the selected model's required sampling constraints here.
applyModelSamplingConstraints(body, this.modelMetadata);

return body;
```

The same invariant should be preserved for the applicable request formats supported by the endpoint. A model's declared sampling constraints should take precedence over generic agent sampling defaults.

This keeps ordinary models unchanged and avoids requiring users to configure global agent settings for a single model.

## Suggested implementation direction

The final request body should be normalized using the selected model's declared constraints. The relevant tests should verify that:

- A model with fixed sampling constraints emits its required `temperature` and `top_p` values when the request supplies different values.
- The behavior covers constraints declared through model metadata or configuration.
- Models without sampling constraints continue to honor explicit per-request sampling values.
- `modelOptions` values continue to work for ordinary models.

The existing custom endpoint tests should be extended rather than introducing a separate test abstraction where possible.

## Validation details

I reproduced this with VS Code's `customendpoint` configuration. Before the fix, Agent mode sent sampling values that were invalid for the selected model. Changing a global agent sampling setting avoids the error, which confirms that the failure is caused by request-option precedence rather than authentication or endpoint selection.

The proposed change should be validated with the focused BYOK endpoint test suite and a manual request using:

- a model with fixed sampling constraints through `customendpoint`
- Agent mode with the default sampling values
- a model without fixed constraints using the same endpoint type

## Related documentation

- [VS Code contribution guide](https://github.com/microsoft/vscode/wiki/How-to-Contribute)
- [BYOK OpenAI endpoint implementation](https://github.com/microsoft/vscode/blob/main/extensions/copilot/src/extension/byok/node/openAIEndpoint.ts)
- [Model capability handling](https://github.com/microsoft/vscode/blob/main/extensions/copilot/src/platform/endpoint/node/chatEndpoint.ts)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.