cache_point parts are silently dropped by every provider — no way to enable prompt caching
- Dominant language
- Zig
- Stars
- 4
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Noticed while porting [coral](https://tangled.sh/@zzstoatzz.io/coral)'s LLM curator to zigai (it now runs the curation loop in production — thanks for this library!).
The message model has `cache_point` parts and usage accounting already handles cache reads/writes (`cache_write_tokens`/`cache_read_tokens`, priced correctly for Anthropic). But on the request side, every provider encoder drops the marker:
- `src/providers/anthropic.zig`: `.cache_point => {}`
- same in `bedrock/converse.zig`, `google.zig`, `openai.zig`, etc.
So there's currently no way to turn prompt caching on: the marker is accepted, encoded to nothing, and the response-side accounting never sees a cache hit.
For Anthropic specifically, the mapping seems mechanical: a `cache_point` part would set `"cache_control": {"type": "ephemeral"}` on the preceding content block (system prompt, tool definitions, or message content). Bedrock's Converse API has the analogous `cachePoint` block.
Motivating example: coral's curator is an agent with a fixed ~2KB system prompt called every 5 minutes — a cache point after the system prompt would cut that portion of input cost to 10% on every call after the first.
Happy to PR the Anthropic side if you'd take it.
---
drafted by claude at my direction, thanks again for the library!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the cache_point handling in src/providers/anthropic.zig, then compare the corresponding encoders in bedrock/converse.zig, google.zig, openai.zig, and the other provider files. Trace the message model and existing cache usage accounting before mapping the marker to each provider's request format. Done means cache_point parts are no longer silently dropped and the applicable provider encoders preserve prompt-caching behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, zig
- Domain
- ai, backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100