github / github/copilot-cli

Enhance the prompt caching for Claude Sonnet model to reduce latency and token costs

オープン
#3,808 コメント 2 件 リアクション 5 件 担当者 0 名 GitHub で見る
area:context-memory area:models
主要言語
Shell
スター
11.2k
フォーク
1.9k
平均マージ
14時間 16分
マージ済み PR(30日)
6

説明

### Describe the feature or problem you'd like to solve

When using GitHub Copilot CLI with the Claude Sonnet model, there is no visible optimization for Anthropic's prompt caching feature. For long system prompts or repeated context (e.g., large codebases, long instruction blocks), each request re-processes the same tokens, leading to higher latency and unnecessary token usage.

### Proposed solution

Leverage Anthropic's prompt caching API (cache_control breakpoints) for static portions of the prompt — such as system instructions, repo context, and tool definitions. This would:
- Reduce time-to-first-token for follow-up turns in the same session
- Lower API costs by reusing cached prefixes (cached tokens are ~90% cheaper)
- Improve responsiveness for users working in large codebases

### Example prompts or workflows

Leverage Anthropic's prompt caching API (cache_control breakpoints) for static portions of the prompt — such as system instructions, repo context, and tool definitions. Specifically:
1. **Cache TTL configuration**: Allow users to configure the cache TTL via a settings option — choosing between the default 5-minute TTL or an extended 1-hour TTL (supported by Anthropic's API), suitable for long working sessions.
2. **Cache visibility CLI command**: Add a command (e.g., `copilot cache status`) to display per-turn cache hit/miss stats using the usage fields already returned by Anthropic's API (`cache_read_input_tokens`, `cache_creation_input_tokens`), helping users understand caching efficiency and debug unexpected misses.

Benefits:
- Reduce time-to-first-token for repeated context in long sessions
- Lower API costs (cached tokens are ~90% cheaper)
- Give power users transparency and control over caching behavior

### Additional context

1. A user runs `copilot cache status` after a multi-turn session and sees that 80% of system prompt tokens were served from cache, confirming cost savings.
2. A user sets cache TTL to 1 hour in config (`copilot config set cache-ttl 1h`) to avoid cache expiry during a long debugging session on a large codebase.
3. A developer asks repeated questions about the same large file — cache hits on the file context reduce response latency from ~3s to ~0.5s after the first turn.
4. A user notices cache misses on every turn via `copilot cache status` and realizes their dynamic timestamp in the system prompt is breaking the cache prefix.
5. A team configures 1h TTL in a shared `.copilot/config` to optimize CI/CD pipelines where the same repo context is queried repeatedly within an hour.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

No files, tests, or entry points are named. Start by locating prompt assembly and the Anthropic request path, then determine how session context and usage fields are represented. Done means static prompt portions use cache breakpoints, TTL is configurable, and a CLI view reports cache hits and misses.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
shell
領域
ai, api, cli, performance
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
静か
明瞭さ
説明が足りない
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。