WeblateOrg / WeblateOrg/weblate

Anthropic machinery: use prompt caching (cache_control) to reduce translation costs

Open Beginner friendly
#21,309 1 comment 0 reactions 0 assignees View on GitHub
Area: Automatic translation good first issue Waiting for: Demand
Dominant language
Python
Stars
6.1k
Forks
1.4k
Avg merge
9h 53m
Merged PRs (30d)
395

Description

### Describe the problem

The Anthropic machinery sends the system prompt as a plain string, without any cache_control block. Anthropic's prompt caching is opt-in, so nothing gets cached at all right now.

The system prompt built in BaseLLMTranslation (persona + style + language instructions + the JSON schema boilerplate) is identical for every request to a given target language. With a longer style guide it easily reaches 2-3k tokens. When automatic translation runs over a few thousand strings, that same prefix is billed at full input price on every single request. In our setup (~3.4k strings per language, ~8KB style prompt) this repeated prefix is the majority of the MT spend.

### Solution brainstorm

Send the system prompt as a content block with a cache breakpoint in weblate/machinery/anthropic.py:

```python
"system": [
{"type": "text", "text": prompt, "cache_control": {"type": "ephemeral"}},
],
```

Cached reads are billed at roughly 10% of the normal input price with a 5 minute sliding TTL, which fits automatic translation nicely since the requests come back to back. The variable parts (glossary, few-shot examples, source strings) are already in the messages after the system prompt, so one breakpoint on the system block is enough.

There is a minimum cacheable prefix (about 1024 tokens) below which the API silently ignores cache_control, so the change should be safe to apply unconditionally.

Docs: https://docs.claude.com/en/docs/build-with-claude/prompt-caching

### Describe alternatives you have considered

Running a small reverse proxy (configured through the machinery base_url setting) that injects cache_control into outgoing requests. It works, but it adds a moving part that Weblate could just handle out of the box with a one line change.

### Screenshots

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in weblate/machinery/anthropic.py and read Anthropic's prompt-caching documentation to confirm the request format and cache limits. Update the system prompt request so it uses a cache breakpoint while leaving the variable glossary, examples, and source strings in the following messages; automatic translation should then reuse the eligible system prompt prefix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, localization
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.