microsoft / microsoft/vscode

Feature Request: Support BYOK / Custom Models for Inline Completions (Ghost Text)

Open
#328,356 2 comments 0 reactions 1 assignee Assigned to @ulugbekna View on GitHub
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

## Summary

Currently, BYOK (Bring Your Own Key) and custom model providers only work for **Chat** (Panel, Inline Chat, Agent Mode) via the `LanguageModelChatProvider` API. **Inline completions (Ghost Text) are hardwired to GitHub's cloud infrastructure** and cannot use custom models.

This creates a critical gap for:
- **Air-gapped environments** (government, defense, critical infrastructure)
- **Data privacy compliance** (GDPR, HIPAA, SOC 2)
- **Cost optimization** (self-hosted models for high-volume completions)
- **Offline development** (no internet connection)

## Current Behavior

1. ✅ Chat works with BYOK models (Ollama, OpenAI-compatible, Anthropic, etc.)
2. ❌ Inline completions **require GitHub Copilot cloud** (no BYOK support)
3. ❌ Setting `github.copilot.selectedCompletionModel` to a custom model ID is **silently ignored**
4. ❌ Disconnecting from the internet **disables completions entirely**

## Proposed Solution

Extend the `LanguageModelChatProvider` API (or introduce a new `LanguageModelCompletionProvider` API) to support custom model providers for inline completions.

### Key Requirements

1. **New API Proposal** (or extend `chatProvider`):
- Add `ChatLocation.Completion` to the `ChatLocation` enum
- Allow `LanguageModelChatProvider` to serve completion requests
- OR: Create `registerLanguageModelCompletionProvider()` API

2. **Prompt Format Support**:
- Completions use a specialized format (prefix/suffix/blockMode/indentation)
- The API should abstract this, or provide a standard completion interface
- Consider OpenAI's `/v1/completions` format as a baseline

3. **Model Validation**:
- Remove the `genericModels` whitelist filter for BYOK providers
- Allow custom model IDs to bypass CAPI validation when registered via BYOK

4. **Configuration**:
- Extend `github.copilot.chat.customOAIModels` (or new setting) to include completion models
- Allow separate model selection for Chat vs. Completions

## Use Cases

### Air-Gapped Government Agency
```jsonc
// .vscode/settings.json
{
"github.copilot.chat.customOAIModels": [
{
"id": "local-coder",
"name": "Local CodeLlama 70B",
"baseUrl": "http://llm-server.internal:8080/v1",
"apiKey": "none",
"model": "codellama-70b",
"capabilities": {
"chat": true,
"completion": true // ← NEW: Support completions
}
}
]
}
```

### Cost-Optimized Enterprise
```jsonc
{
"github.copilot.chat.customOAIModels": [
{
"id": "deepseek-coder",
"name": "DeepSeek Coder (Self-hosted)",
"baseUrl": "http://ai-pool.internal:8080/v1",
"capabilities": {
"chat": false, // Use Copilot for chat (complex reasoning)
"completion": true // Use self-hosted for completions (high-volume)
}
}
]
}
```

## Acceptance Criteria

- [ ] Custom models can be registered for inline completions via API
- [ ] `ChatLocation` enum includes a `Completion` value (or equivalent)
- [ ] `genericModels` whitelist is bypassed for BYOK-registered completion models
- [ ] Completion prompt format is abstracted or documented for custom providers
- [ ] Configuration allows separate model selection for Chat vs. Completions
- [ ] Air-gapped environments can use completions without internet connectivity
- [ ] No regression for existing Copilot completion behavior

## Related Issues

- #318545 - Support BYOK/Custom Models for Inline Code Completions (156+ upvotes)
- #246551 - Use copilot with local models completely offline (229+ upvotes)
- #246303 - Add CoPilot configuration option for OpenAI compatible endpoints

## Environment

- VS Code Version: 1.104+
- OS: All (Windows, macOS, Linux)
- Copilot Extension: Latest

## Additional Context

The `LanguageModelChatProvider` API already supports BYOK for Chat locations (Agent, Terminal, Notebook, InlineChat). Extending this to Completions would be a natural evolution of the existing API, maintaining consistency and reducing implementation complexity.

The technical infrastructure exists — the `byok/` directory in the Copilot extension already contains implementations for Anthropic, Gemini, XAI, Azure, and custom OpenAI-compatible providers. These just need to be wired into the completions pipeline.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.