github / github/copilot-cli

Enhance the prompt caching for Claude Sonnet model to reduce latency and token costs

未关闭
#3,808 2 条评论 5 个 reaction 已指派 0 人 在 GitHub 查看
area:context-memory area:models
主要语言
Shell
星标
11.2k
派生
1.9k
平均合并
14 小时 16 分钟
30 天内合并 PR
6

描述

### Describe the feature or problem you'd like to solve

When using GitHub Copilot CLI with the Claude Sonnet model, there is no visible optimization for Anthropic's prompt caching feature. For long system prompts or repeated context (e.g., large codebases, long instruction blocks), each request re-processes the same tokens, leading to higher latency and unnecessary token usage.

### Proposed solution

Leverage Anthropic's prompt caching API (cache_control breakpoints) for static portions of the prompt — such as system instructions, repo context, and tool definitions. This would:
- Reduce time-to-first-token for follow-up turns in the same session
- Lower API costs by reusing cached prefixes (cached tokens are ~90% cheaper)
- Improve responsiveness for users working in large codebases

### Example prompts or workflows

Leverage Anthropic's prompt caching API (cache_control breakpoints) for static portions of the prompt — such as system instructions, repo context, and tool definitions. Specifically:
1. **Cache TTL configuration**: Allow users to configure the cache TTL via a settings option — choosing between the default 5-minute TTL or an extended 1-hour TTL (supported by Anthropic's API), suitable for long working sessions.
2. **Cache visibility CLI command**: Add a command (e.g., `copilot cache status`) to display per-turn cache hit/miss stats using the usage fields already returned by Anthropic's API (`cache_read_input_tokens`, `cache_creation_input_tokens`), helping users understand caching efficiency and debug unexpected misses.

Benefits:
- Reduce time-to-first-token for repeated context in long sessions
- Lower API costs (cached tokens are ~90% cheaper)
- Give power users transparency and control over caching behavior

### Additional context

1. A user runs `copilot cache status` after a multi-turn session and sees that 80% of system prompt tokens were served from cache, confirming cost savings.
2. A user sets cache TTL to 1 hour in config (`copilot config set cache-ttl 1h`) to avoid cache expiry during a long debugging session on a large codebase.
3. A developer asks repeated questions about the same large file — cache hits on the file context reduce response latency from ~3s to ~0.5s after the first turn.
4. A user notices cache misses on every turn via `copilot cache status` and realizes their dynamic timestamp in the system prompt is breaking the cache prefix.
5. A team configures 1h TTL in a shared `.copilot/config` to optimize CI/CD pipelines where the same repo context is queried repeatedly within an hour.

贡献指南

打开贡献指南

调研方向

No files, tests, or entry points are named. Start by locating prompt assembly and the Anthropic request path, then determine how session context and usage fields are represented. Done means static prompt portions use cache breakpoints, TTL is configurable, and a CLI view reports cache hits and misses.

由索引模型根据 Issue 内容生成。

评估

技术栈
shell
领域
ai, api, cli, performance
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
冷清
描述清晰度
需要澄清
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。