charmbracelet / charmbracelet/fantasy

Agent API: no way to place a cache_control breakpoint inside the system prompt

Open
#305 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
987
Forks
137
Avg merge
14h 41m
Merged PRs (30d)
12

Description

### Summary

The Anthropic provider already supports per-part `cache_control` everywhere — including system blocks: `toPrompt` in `providers/anthropic/anthropic.go` reads `GetCacheControl(part.Options())` for each system `TextPart` (falling back to `msg.ProviderOptions` on the last part), so a multi-part system message can pin a cache breakpoint exactly where the stable prefix ends.

But that capability is unreachable through the Agent API: `agentSettings.systemPrompt` is a plain `string` (`agent.go`), so the system prompt always becomes a single text part with no way to attach per-part provider options. The only lever available on an `AgentCall`/`AgentStreamCall` is call-level `ProviderOptions`, which can't express "cache the first N% of the system prompt, keep the tail volatile."

### Why it matters

The classic Anthropic prompt-caching layout is a large stable system prefix (instructions, rubric, reference material) plus a small volatile suffix (recent context). Apps with that shape currently have to choose between the Agent conveniences and effective prefix caching — or drop down to the `LanguageModel` layer and build the prompt by hand.

### Suggested shape (whatever fits the API's direction)

- `agent.WithSystemPromptParts(parts ...fantasy.TextPart)` (or accept a pre-built system `Message`), letting callers attach `ProviderOptions`/part options carrying `anthropic.ProviderCacheControlOptions` — the provider plumbing already honors them; or
- a lighter `WithSystemPromptCacheBoundary(offset int)` style hint the Anthropic provider translates into a two-block system message.

Related precedent: #302 added per-message cache markers for OpenAI-compatible providers — this is the same need aimed at the system prompt through the Agent path.

Happy to take a swing at a PR if one of these directions sounds right to you.

Contributor guide

Open the contributing guide

Research direction

Read agent.go to trace how agentSettings.systemPrompt becomes a system message, then inspect providers/anthropic/anthropic.go and its toPrompt handling of system TextPart options. Compare the proposed Agent API shapes with the per-message cache markers from #302. Done means the Agent path can express a cache boundary inside a system prompt while preserving the provider's existing per-part cache_control behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.