microsoft / microsoft/vscode

Better Custom Endpoint BYOK

Open
#325,237 10 comments 26 reactions 1 assignee Claimed by @vritant24 View on GitHub
feature-request model-byok
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

1. Add auto-fetch models and most model settings.
2. Allow configuring Base URL for only one times.
3. Allow omitting most of the model configs.
4. Allow global model settings.
5. Add detailed information: icons, descriptions, total CWS, credits, and more.
See the example configuration file below for more information.

```jsonc
[
{
"name": "Custom Endpoint",
"vendor": "customendpoint",
"url": "https://example.com/api",
"apiType": "chat-completions",
/*
1. Allow configuring url outside `models`.
2. Automatically (responses > messages > chat-completions) use the available API type when `apiType` is not provided.
2. Allow full URL (e.g. https://example.com/v1/chat/completions). When using full URL, auto-detect `apiType` and ignore user's config.
*/
"autoFetch": true, // Add: Auto-fetch. If not configured, defaultly enable (only) when `models` is empty.
"global": { // Add: Global settings (optional). Can be overrided by per-model settings below.
"description": "A model with 1M context windows size and vision.", // Add: Model description. Shown when the cursor hovers on the model name in the picker.
"icon": "https://example.com/model.svg", // Add: Model icon. Shown in model picker. Available options: Icon URL, "claude", "gpt", "gemini", etc.
"toolCalling": true,
"vision": true,
"maxInputTokens": 1024000,
"maxOutputTokens": 64000,
"maxTokens": 1024000, // Add: Total context window size. Use the smaller one of configured CWS here and (`maxInputTokens` + `maxOutputTokens`).
"credits": {
// Add: Credits per 1M tokens.
"input": 1,
"output": 3,
"cacheRead": 0.1
}
},
"models": [
{
"id": "custom",
// Everything below is optional. If not provided, automatically fetch the information below if possible.
"name": "My Model",
"description": "A model with 1M context windows size and vision.",
"icon": "https://example.com/model.svg",
"toolCalling": true,
"vision": true,
"maxInputTokens": 1024000,
"maxOutputTokens": 64000,
"maxTokens": 1024000,
"credits": {
"input": 1,
"output": 3,
"cacheRead": 0.1
}
}
] // The model list itself is optional too--auto-fetch if empty.
}
]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.