microsoft / microsoft/vscode

Custom chat model settings collide when multiple providers expose the same model id

Open
#325,069 0 comments 0 reactions 1 assignee Claimed by @vritant24 View on GitHub
bug model-byok
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

# Custom chat model settings collide when multiple providers expose the same model id

## Description

When configuring multiple custom chat language model providers, model-specific settings appear to be keyed only by the model `id`. This causes settings to collide when different providers expose models with the same `id`.

For example, I have two custom providers. Both define a model with `"id": "gpt-5.5"`.

Provider A:

```json
{
"name": "Custom Provider A",
"vendor": "customendpoint",
"apiType": "messages",
"models": [
{
"id": "gpt-5.5",
"name": "GPT-5.5",
"url": "http://127.0.0.1:15721",
"toolCalling": true,
"vision": true,
"streaming": true,
"thinking": true,
"reasoningEffortFormat": "responses",
"supportsReasoningEffort": [
"none",
"minimal",
"low",
"medium",
"high",
"xhigh"
]
}
],
"settings": {
"gpt-5.5": {
"reasoningEffort": "high"
}
}
}
```

Provider B:

```json
{
"name": "Custom Provider B",
"vendor": "customendpoint",
"apiType": "responses",
"models": [
{
"id": "gpt-5.5",
"name": "GPT-5.5-B",
"url": "https://example.com",
"toolCalling": true,
"vision": true,
"streaming": true,
"thinking": true,
"reasoningEffortFormat": "responses",
"supportsReasoningEffort": [
"none",
"minimal",
"low",
"medium",
"high",
"xhigh"
]
}
]
}
```

After selecting `reasoningEffort = high` for Provider A's `gpt-5.5` model, VS Code writes the setting under:

```json
"settings": {
"gpt-5.5": {
"reasoningEffort": "high"
}
}
```

However, if I then select Provider B's `gpt-5.5` model and change `reasoningEffort` to `low`, the existing setting for Provider A is modified instead of creating or updating a setting scoped to Provider B.

## Steps to Reproduce

1. Configure multiple custom chat language model providers.
2. In both providers, define a model with the same `id`, for example `"gpt-5.5"`.
3. Select the model from Provider A and set `reasoningEffort` to `high`.
4. Select the model from Provider B and set `reasoningEffort` to `low`.
5. Inspect the persisted `chatLanguageModels.json` configuration.

## Actual Behavior

The persisted setting is keyed only by model id:

```json
"settings": {
"gpt-5.5": {
"reasoningEffort": "low"
}
}
```

This makes the setting ambiguous and causes one provider's model setting to overwrite another provider's model setting when both models share the same `id`.

## Expected Behavior

Model settings should be scoped by a unique model identity that includes the provider identity, not only the model `id`.

For example, settings could be keyed by a provider-scoped identifier such as:

```json
"settings": {
"Custom Provider A/gpt-5.5": {
"reasoningEffort": "high"
},
"Custom Provider B/gpt-5.5": {
"reasoningEffort": "low"
}
}
```

or stored inside each provider entry so that model settings are local to that provider.

## Why This Matters

It is common for different providers or custom endpoints to expose models with the same model id, especially OpenAI-compatible or gateway-style endpoints. The current behavior makes it unclear which provider/model is affected by a setting and can silently apply the wrong reasoning effort to another provider's model.

## Additional Notes

This appears to be caused by using only `model.id` as the settings key. Since `model.id` is not globally unique across providers, the configuration should probably use a provider-qualified key or persist settings under the corresponding provider object.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.