Kludex / Kludex/zigai

cache_point parts are silently dropped by every provider — no way to enable prompt caching

Open
#1 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Zig
Stars
4
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Noticed while porting [coral](https://tangled.sh/@zzstoatzz.io/coral)'s LLM curator to zigai (it now runs the curation loop in production — thanks for this library!).

The message model has `cache_point` parts and usage accounting already handles cache reads/writes (`cache_write_tokens`/`cache_read_tokens`, priced correctly for Anthropic). But on the request side, every provider encoder drops the marker:

- `src/providers/anthropic.zig`: `.cache_point => {}`
- same in `bedrock/converse.zig`, `google.zig`, `openai.zig`, etc.

So there's currently no way to turn prompt caching on: the marker is accepted, encoded to nothing, and the response-side accounting never sees a cache hit.

For Anthropic specifically, the mapping seems mechanical: a `cache_point` part would set `"cache_control": {"type": "ephemeral"}` on the preceding content block (system prompt, tool definitions, or message content). Bedrock's Converse API has the analogous `cachePoint` block.

Motivating example: coral's curator is an agent with a fixed ~2KB system prompt called every 5 minutes — a cache point after the system prompt would cut that portion of input cost to 10% on every call after the first.

Happy to PR the Anthropic side if you'd take it.

---

drafted by claude at my direction, thanks again for the library!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the cache_point handling in src/providers/anthropic.zig, then compare the corresponding encoders in bedrock/converse.zig, google.zig, openai.zig, and the other provider files. Trace the message model and existing cache usage accounting before mapping the marker to each provider's request format. Done means cache_point parts are no longer silently dropped and the applicable provider encoders preserve prompt-caching behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, zig
Domain
ai, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.