WeblateOrg / WeblateOrg/weblate
Anthropic machinery: use prompt caching (cache_control) to reduce translation costs
- Dominant language
- Python
- Stars
- 6.1k
- Forks
- 1.4k
- Avg merge
- 9h 53m
- Merged PRs (30d)
- 395
Description
### Describe the problem
The Anthropic machinery sends the system prompt as a plain string, without any cache_control block. Anthropic's prompt caching is opt-in, so nothing gets cached at all right now.
The system prompt built in BaseLLMTranslation (persona + style + language instructions + the JSON schema boilerplate) is identical for every request to a given target language. With a longer style guide it easily reaches 2-3k tokens. When automatic translation runs over a few thousand strings, that same prefix is billed at full input price on every single request. In our setup (~3.4k strings per language, ~8KB style prompt) this repeated prefix is the majority of the MT spend.
### Solution brainstorm
Send the system prompt as a content block with a cache breakpoint in weblate/machinery/anthropic.py:
```python
"system": [
{"type": "text", "text": prompt, "cache_control": {"type": "ephemeral"}},
],
```
Cached reads are billed at roughly 10% of the normal input price with a 5 minute sliding TTL, which fits automatic translation nicely since the requests come back to back. The variable parts (glossary, few-shot examples, source strings) are already in the messages after the system prompt, so one breakpoint on the system block is enough.
There is a minimum cacheable prefix (about 1024 tokens) below which the API silently ignores cache_control, so the change should be safe to apply unconditionally.
Docs: https://docs.claude.com/en/docs/build-with-claude/prompt-caching
### Describe alternatives you have considered
Running a small reverse proxy (configured through the machinery base_url setting) that injects cache_control into outgoing requests. It works, but it adds a moving part that Weblate could just handle out of the box with a one line change.
### Screenshots
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start in weblate/machinery/anthropic.py and read Anthropic's prompt-caching documentation to confirm the request format and cache limits. Update the system prompt request so it uses a cache breakpoint while leaving the variable glossary, examples, and source strings in the following messages; automatic translation should then reuse the eligible system prompt prefix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, localization
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100