Enhance the prompt caching for Claude Sonnet model to reduce latency and token costs
- Lenguaje dominante
- Shell
- Estrellas
- 11.2k
- Forks
- 1.9k
- Merge medio
- 14 h 16 min
- PR fusionados (30 d)
- 6
Descripción
### Describe the feature or problem you'd like to solve
When using GitHub Copilot CLI with the Claude Sonnet model, there is no visible optimization for Anthropic's prompt caching feature. For long system prompts or repeated context (e.g., large codebases, long instruction blocks), each request re-processes the same tokens, leading to higher latency and unnecessary token usage.
### Proposed solution
Leverage Anthropic's prompt caching API (cache_control breakpoints) for static portions of the prompt — such as system instructions, repo context, and tool definitions. This would:
- Reduce time-to-first-token for follow-up turns in the same session
- Lower API costs by reusing cached prefixes (cached tokens are ~90% cheaper)
- Improve responsiveness for users working in large codebases
### Example prompts or workflows
Leverage Anthropic's prompt caching API (cache_control breakpoints) for static portions of the prompt — such as system instructions, repo context, and tool definitions. Specifically:
1. **Cache TTL configuration**: Allow users to configure the cache TTL via a settings option — choosing between the default 5-minute TTL or an extended 1-hour TTL (supported by Anthropic's API), suitable for long working sessions.
2. **Cache visibility CLI command**: Add a command (e.g., `copilot cache status`) to display per-turn cache hit/miss stats using the usage fields already returned by Anthropic's API (`cache_read_input_tokens`, `cache_creation_input_tokens`), helping users understand caching efficiency and debug unexpected misses.
Benefits:
- Reduce time-to-first-token for repeated context in long sessions
- Lower API costs (cached tokens are ~90% cheaper)
- Give power users transparency and control over caching behavior
### Additional context
1. A user runs `copilot cache status` after a multi-turn session and sees that 80% of system prompt tokens were served from cache, confirming cost savings.
2. A user sets cache TTL to 1 hour in config (`copilot config set cache-ttl 1h`) to avoid cache expiry during a long debugging session on a large codebase.
3. A developer asks repeated questions about the same large file — cache hits on the file context reduce response latency from ~3s to ~0.5s after the first turn.
4. A user notices cache misses on every turn via `copilot cache status` and realizes their dynamic timestamp in the system prompt is breaking the cache prefix.
5. A team configures 1h TTL in a shared `.copilot/config` to optimize CI/CD pipelines where the same repo context is queried repeatedly within an hour.
Guía de contribución
Línea de trabajo
No files, tests, or entry points are named. Start by locating prompt assembly and the Anthropic request path, then determine how session context and usage fields are represented. Done means static prompt portions use cache breakpoints, TTL is configurable, and a CLI view reports cache hits and misses.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- shell
- Área
- ai, api, cli, performance
- Tipo de issue
- Nueva funcionalidad
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Estado de actividad
- Tranquilo
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 35/100