github / github/copilot-cli

Enhance the prompt caching for Claude Sonnet model to reduce latency and token costs

Abierto
#3,808 2 comentarios 5 reacciones 0 asignados Ver en GitHub
area:context-memory area:models
Lenguaje dominante
Shell
Estrellas
11.2k
Forks
1.9k
Merge medio
14 h 16 min
PR fusionados (30 d)
6

Descripción

### Describe the feature or problem you'd like to solve

When using GitHub Copilot CLI with the Claude Sonnet model, there is no visible optimization for Anthropic's prompt caching feature. For long system prompts or repeated context (e.g., large codebases, long instruction blocks), each request re-processes the same tokens, leading to higher latency and unnecessary token usage.

### Proposed solution

Leverage Anthropic's prompt caching API (cache_control breakpoints) for static portions of the prompt — such as system instructions, repo context, and tool definitions. This would:
- Reduce time-to-first-token for follow-up turns in the same session
- Lower API costs by reusing cached prefixes (cached tokens are ~90% cheaper)
- Improve responsiveness for users working in large codebases

### Example prompts or workflows

Leverage Anthropic's prompt caching API (cache_control breakpoints) for static portions of the prompt — such as system instructions, repo context, and tool definitions. Specifically:
1. **Cache TTL configuration**: Allow users to configure the cache TTL via a settings option — choosing between the default 5-minute TTL or an extended 1-hour TTL (supported by Anthropic's API), suitable for long working sessions.
2. **Cache visibility CLI command**: Add a command (e.g., `copilot cache status`) to display per-turn cache hit/miss stats using the usage fields already returned by Anthropic's API (`cache_read_input_tokens`, `cache_creation_input_tokens`), helping users understand caching efficiency and debug unexpected misses.

Benefits:
- Reduce time-to-first-token for repeated context in long sessions
- Lower API costs (cached tokens are ~90% cheaper)
- Give power users transparency and control over caching behavior

### Additional context

1. A user runs `copilot cache status` after a multi-turn session and sees that 80% of system prompt tokens were served from cache, confirming cost savings.
2. A user sets cache TTL to 1 hour in config (`copilot config set cache-ttl 1h`) to avoid cache expiry during a long debugging session on a large codebase.
3. A developer asks repeated questions about the same large file — cache hits on the file context reduce response latency from ~3s to ~0.5s after the first turn.
4. A user notices cache misses on every turn via `copilot cache status` and realizes their dynamic timestamp in the system prompt is breaking the cache prefix.
5. A team configures 1h TTL in a shared `.copilot/config` to optimize CI/CD pipelines where the same repo context is queried repeatedly within an hour.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

No files, tests, or entry points are named. Start by locating prompt assembly and the Anthropic request path, then determine how session context and usage fields are represented. Done means static prompt portions use cache breakpoints, TTL is configurable, and a CLI view reports cache hits and misses.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
shell
Área
ai, api, cli, performance
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Tranquilo
Claridad
Necesita aclaración
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.