Machine translation decodes HTML entities in source strings: `&lt;script&gt;` is stored as `<script>`, or deleted outright when `safe-html` is set

### Describe the issue

For the engines that mix in `XMLMachineTranslationMixin` (`deepl`, `google-translate-api-v3`, `microsoft-translator`), Weblate wraps every machinery call in an escape/unescape pair:

* `cleanup_text()` applies `escape_text()` → `html.escape()` (`weblate/machinery/base.py:486-497`, `:1296`)
* `uncleanup_text()` applies `unescape_text()` → `html.unescape()` (`weblate/machinery/base.py:511`, `:1292`)

That round-trip is only lossless if the provider returns the escaping **at the level Weblate sent it**. HTML-mode providers do not guarantee that, and in practice they renormalise. When they do, Weblate's unconditional unescape on the return path removes one level of escaping that was never re-added, and a source string containing a literal HTML entity comes back decoded.

Observed on a source string whose *correct* translation is inert text:

```
source To disable it, write <script> in the template.
cleaned To disable it, write &lt;script&gt; in the template. <- Weblate escapes
reply Um es zu deaktivieren, schreiben Sie <script> in die Vorlage. <- provider returns one level decoded
stored Um es zu deaktivieren, schreiben Sie in die Vorlage. <- Weblate unescapes again
```

Inert, escaped text in the source has become live markup in the target.

Whether the provider *should* have returned `&amp;lt;` rather than `&lt;` is arguable, and this report does not rest on it. What is not arguable is where the damage happens: **at the moment Weblate receives the reply the content is still an escaped entity, and therefore still inert. Weblate performs the final decode that turns it into live markup.** The defect is that `uncleanup_text()` unescapes unconditionally, assuming the escaping level it sent survives the round trip, with no check that it did.

That the provider's reply is still **escaped, inert and recoverable** at the moment Weblate receives it can be shown directly, because the value is cached before the unescape is applied. (Not that it is the *correct* translation — the provider has already normalised one escaping level. The point is narrower: what arrives is still inert text, and what Weblate does to it next is what makes it dangerous.) Instrumenting a single `google-translate-api-v3` call through the machinery layer:

```
cached provider result Um es zu deaktivieren, schreiben Sie &lt;script&gt; in die Vorlage.
returned to the caller Um es zu deaktivieren, schreiben Sie <script> in die Vorlage.
```

The cache entry and the returned value come from the same single provider call (`provider_calls_first: 1`). A second `translate()` with the network hard-blocked returns the corrupted form again from that same intact cache entry, so the corruption is applied on **every read**, not once at fetch time. Raw data: `round2-gap-probe.json → translation_cache`.

**With the `safe-html` flag set, this turns into silent data loss.** The `BleachHTML` autofix (`weblate/trans/autofixes/html.py`, present in `DEFAULT_AUTOFIX_LIST`) runs on save, sees the now-live `<script>`, and `nh3.clean(..., clean_content_tags={"script","style"} - tags)` removes the element *and everything after it*:

```
source To disable it, write &lt;script&gt; in the template.
stored Um es zu deaktivieren, schreiben Sie
```

The rest of the sentence is gone, and it is gone from the translation file on disk, not just from the UI. Forcing a repository commit and reading the working copy out of the container confirms the database and the file agree:

```
POST /api/components/{p}/{c}/repository/ {"operation": "commit"} -> 200

database file on disk
/app/data/vcs/.../json-safe/de.json
escaped_script "Um es zu deaktivieren, ..." "Um es zu deaktivieren, schreiben Sie "
escaped_tag "Das -Tag kennzeichnet ..." "Das -Tag kennzeichnet Text als fett."
```

`<b>` is removed the same way (`The &lt;b&gt; tag makes text bold.` → `Das -Tag kennzeichnet Text als fett.`), because `extract_html_tags()` is given the *source*, which contains no real tags — only entities — so the allowed-tag set is empty and `clean_content_tags=CLEAN_CONTENT_TAGS - tags` therefore covers everything:

```python
# weblate/utils/html.py:345-360 HTMLSanitizer.clean
tags, attributes = extract_html_tags(source)
text = nh3.clean(text, link_rel=None, tags=tags, attributes=attributes,
clean_content_tags=CLEAN_CONTENT_TAGS - tags)
```

`weblate.trans.autofixes.html.BleachHTML` is active in a default installation — confirmed present in `settings.AUTOFIX_LIST` on the instance used here. Raw data: `ondisk-loss-verify.json`.

In every case `autotranslate` returns `200 {"details": "Automatic translation completed, 4 strings were updated."}` and **no check fires on the resulting unit**. There is no signal to the user or to automation that anything was lost.

The provider is inconsistent even within a single string, so this cannot be compensated for by assuming a fixed escaping level. German, `google-translate-api-v3`, one request:

```
cleaned Escape &amp;lt;, &amp;gt; and &amp;amp; before saving.
reply Vor dem Speichern die Escape-Taste (&lt;, &amp;gt; und &amp;amp;) drücken.
stored Vor dem Speichern die Escape-Taste (<, &gt; und &amp;) drücken.
```

The first entity came back single-escaped and was corrupted; the other two came back double-escaped and survived.

Across a deliberately entity-heavy set of 7 sources × 4 locales, one attempt each, for the three engines that use the HTML/XML mixin:

| Engine | intact | decoded | became live markup |
|---|---|---|---|
| `deepl` | 21/28 | 7/28 | 0/28 |
| `google-translate-api-v3` | 10/28 | 10/28 | **8/28** |
| `microsoft-translator` | 1/28 | 26/28 | 1/28 |

Repeating every call showed the outcome is deterministic (110 of 112 repeat pairs byte-identical; the 2 exceptions were DeepL wording variation that preserved the entities in both attempts).

**Please read those ratios as "this is reproducible on demand", not as a corruption rate for real projects.** The corpus was built to concentrate literal entities, so the denominator is not representative of ordinary translation content. The claim is that the outcome is deterministic per string, not that a given share of any real project is affected.

(Weblate applies no escaping at all to the `google-translate` v2 engine, which sends `format=text`, so it does not traverse this code path. Entity-bearing sources are also mangled there, but by a different mechanism, and it is not evidence about this one.)

**It is not a file-format problem — and Weblate already has a mechanism that prevents it.**

The same four source strings, the same engine, the same locale, uploaded as four different file formats:

| Format | without `safe-html` | with `safe-html` |
|---|---|---|
| JSON | 3/4 corrupted | 2/4 corrupted |
| Android resources | 3/4 corrupted | 2/4 corrupted |
| gettext PO | 3/4 corrupted | 2/4 corrupted |
| XLIFF 1.2 | **0/4** | 2/4 corrupted |

JSON, Android and gettext produce **byte-identical** target strings — three unrelated serialisations, same output — which places the defect in the machinery layer, not in any format's parser or writer.

XLIFF is the interesting one, because it shows a mechanism that prevents this already exists in the codebase. XLIFF units carry an automatic `xml-text` flag (`RichXliffUnit.add_flags`), and with that flag Weblate highlights the entity references themselves and protects them as non-translatable spans before the call:

```
source To disable it, write &lt;script&gt; in the template.

no flags on wire To disable it, write &amp;lt;script&amp;gt; in the template.
highlights: [] -> corrupted

xml-text on wire To disable it, write <span translate="no" id="21">&amp;amp;lt;</span>script<span translate="no" id="31">&amp;amp;gt;</span> in the template.
highlights: [(21,25,'&lt;'), (31,35,'&gt;')] -> survives
```

So the escape/unescape round trip is only lossy for entity spans that were never protected. `xml-text` protects them and the corruption disappears; every format without that flag is exposed. Raw data: `xliff-flag-probe.json`, `entity-format-matrix.json`.

Two caveats, so this is not overstated. Protection preserves the entity but not the surrounding whitespace — the XLIFF targets came back as `&lt; script &gt;`, with spaces the provider inserted around the protected spans. And `safe-html` defeats the protection completely: with that flag set, XLIFF's targets are byte-identical to JSON's, so all four formats converge on the same corrupted output.

`checks` was empty on **every** unit in all eight components, corrupted or not.

### I already tried
- [x] I've read and searched the documentation.
- [x] I've searched for similar filed issues in this repository.

(Searched for prior art on entity handling, double escaping, `escape_text`/`unescape_text`, and `safe-html` removing translation content.)

Nearest existing items, none of which cover this:

* **#12936 / #12938** introduced the forced uncleanup that this report is about. It was the right fix for its case — raw ampersands coming back escaped — and its regression coverage is written around that case, not around source text that already contained a literal entity before translation. Any fix here needs to keep #12936 fixed.
* **#17048 / #17050** fixed a different defect in the same restoration function (`re.sub` interpreting escape sequences in the replacement string) by switching to a callable replacement. It made restoration stop corrupting the replacement; it does not touch the escaping level.
* **Discussion #6657** and **#11995** concern entity display and file-format handling, not the machinery round trip.
* **#6478** proposed an entity-count check. That would be a useful oracle for exactly this failure, but it was closed as stale.
* **#6958** concerns the inverse DeepL behaviour (entities being introduced), not entities being decoded.

Most relevant of all, and the reason I think the `safe-html` half of this is not controversial:

* **#18967** — *"Entering `<` in a translation clears the translation field and saves an empty string"*. Same end state as here (the `safe-html` autofix destroying translated content), reached by a different route: a human typing into an Android resource unit rather than machine translation writing to it. It was accepted as a defect and fixed by **PR #18997**, at the file-format boundary. The machinery path is untouched by that fix, which is why this is adjacent rather than duplicate — but the principle stated in it applies directly here: *"Under no circumstances should Weblate clear text the translator has entered."* Machine translation output that a reviewer is about to see deserves the same guarantee.
* **#17093** — *"Erroneous 'XML tags in translation do not match source' error when using entities"*, closed as not planned. Adjacent entity-fidelity concern on the human-editing path, in the opposite direction: a check firing when it should not, rather than content vanishing with no check at all.

I could not find any existing report covering the machinery escape/unescape path specifically.

### Steps to reproduce the behavior

A self-contained script using only the public REST API is attached below. Manually:

1. Configure `google-translate-api-v3`.
2. Create a JSON component with `check_flags` set to `safe-html`.
3. Add a source string containing a literal escaped tag, e.g.
`To disable it, write &lt;script&gt; in the template.`
4. Add a `de` translation and run
`POST /api/translations/{p}/{c}/de/autotranslate/` with
`{"mode": "translate", "q": "state:empty", "auto_source": "mt", "engines": ["google-translate-api-v3"], "threshold": 10}`.
5. Read the unit back. The target is `Um es zu deaktivieren, schreiben Sie ` — everything from the escaped tag onwards has been removed. `checks` is empty.
6. Repeat with `check_flags` empty. The target is now
`Um es zu deaktivieren, schreiben Sie <script> in die Vorlage.` — the escaped tag has been promoted to live markup instead of deleted.

Reproduction script:

```python
# repro-mt-entity-roundtrip.py -- public REST API only, creates and deletes a
# disposable project.
# export WLTOKEN=<admin token>; export WEBLATE_URL=http://localhost:8080
# python3 repro-mt-entity-roundtrip.py # safe-html
# python3 repro-mt-entity-roundtrip.py --no-safe-html # without the flag
```

Output on 2026.7.1, `safe-html`, 3 locales × 4 strings → 6/12 units corrupted:

```
=== de: autotranslate HTTP 200 {'details': 'Automatic translation completed, 4 strings were updated.'}
escaped_script CORRUPTED checks=[]
source: 'To disable it, write &lt;script&gt; in the template.'
target: 'Um es zu deaktivieren, schreiben Sie '
escaped_tag CORRUPTED checks=[]
source: 'The &lt;b&gt; tag makes text bold.'
target: 'Das -Tag kennzeichnet Text als fett.'
```

Without `safe-html`, 7/12 corrupted:

```
escaped_script CORRUPTED checks=[]
source: 'To disable it, write &lt;script&gt; in the template.'
target: 'Um es zu deaktivieren, schreiben Sie <script> in die Vorlage.'
```

### Expected behavior

A source string containing a literal HTML entity should round-trip through machine translation with its escaping level intact — `&lt;script&gt;` in the source should stay `&lt;script&gt;` in the target, not become `<script>` and not disappear.

Concretely, any of:

* **Shield entity spans instead of relying on double-escaping, building on the mechanism that already exists.** Under `xml-text`, Weblate already highlights entity references and protects them as non-translatable spans, and units with that flag do not exhibit this corruption. Extending that protection to entity spans generally, independent of the flag, would address this without inventing a new mechanism. To be clear this is a starting point rather than a finished fix: `xml-text` protection is not lossless (the provider inserted whitespace around the protected spans) and `safe-html` defeats it entirely, so it demonstrates feasibility, not a drop-in solution. This is deliberately *not* a proposal to drop the unconditional unescape — that unescape is what fixes #12936, and removing it would regress that. The narrower change is to stop pre-existing entity spans from entering the escape/unescape pair at all.
* Detect the mismatch rather than absorb it. If the provider's reply does not contain the escaping level that was sent, that is a signal the round-trip failed — surfacing it as a check or a machinery error would at least make the loss visible instead of silent. #6478's entity-count idea would serve here.
* At minimum, make the `safe-html` autofix not delete content that machine translation itself introduced in the same operation. Removing an entire clause with no check, no warning and a success-shaped API response is the part that turns a quality problem into a data-loss problem.

### Screenshots

N/A

### Exception traceback

No exception is raised — that is central to the report. Every call returns `200` and reports success.

### How do you run Weblate?

Docker container

### Weblate version

2026.7.1 (image `weblate/weblate:2026.7.1.1`)

Still present on `main` (verified at `8a20ef28`): `cleanup_text`, `uncleanup_text`, `uncleanup_text_item`, `escape_text`, `unescape_text` and `make_re_placeholder` are byte-identical to the tested tag; `force_uncleanup` and the `uncleanup_results` call are unchanged apart from being relocated into a new `_apply_downloaded_translations` helper, which still writes the provider result to cache before uncleanup runs. `weblate/checks/placeholders.py`, `weblate/machinery/googlev3.py` and `weblate/trans/autofixes/html.py` are byte-identical. That is a method-level comparison, not a claim about everything else that changed in the intervening commits.

### Weblate deploy checks

N/A — reproduced on a minimal, disposable single-node instance for isolation.

### Additional context

Found while building an edge-case corpus for machine translation. Note that an earlier round of this work measured provider APIs directly and drew the opposite conclusion — that Weblate fails to unescape Google v3 output. That was wrong: `XMLMachineTranslationMixin.unescape_text()` does unescape, and with `force_uncleanup = True` it runs on every result. The actual defect is the reverse — Weblate unescapes output that the provider had already decoded once.

Happy to share the full instrumented transcripts (source → `cleanup_text` output → provider reply → stored value, for 5 engines across 4 locales) if useful for a regression test.

Start with cleanup_text() and uncleanup_text() in weblate/machinery/base.py, then inspect XMLMachineTranslationMixin and the BleachHTML path in weblate/trans/autofixes/html.py. Reproduce the public API case with google-translate-api-v3 and compare it with the xml-text protection described for XLIFF. Done means entity-bearing source text remains inert, safe-html does not delete it, and the existing behavior covered by #12936 remains intact.

Written by the indexing model from the issue text.