dgtlmoon / dgtlmoon/changedetection.io
[i18n] Consolidate fragmented gettext calls into entire-sentence msgids
- Dominant language
- Python
- Stars
- 34.1k
- Forks
- 2k
- Avg merge
- 23h 38m
- Merged PRs (30d)
- 67
Description
## tl;dr
Many template locations split a single English sentence across multiple `{{ _('...') }}` fragments, and this is degrading translation quality.
Following the GNU gettext guidelines, I'd like to adopt a policy of **not fragmenting msgids — keeping each at the sentence (or paragraph) level**.
## Why this is a problem
### Problem 1: For many non-English languages, fragment splitting prevents natural translation, and the existing workaround breaks on reuse
#### Why fragments break
Because gender/number/case agreement and word order depend on the entire-sentence context
(as the gettext manual's [Entire sentences](https://www.gnu.org/software/gettext/manual/html_node/Entire-sentences.html) and [No string concatenation](https://www.gnu.org/software/gettext/manual/html_node/No-string-concatenation.html) pages point out
— the former focuses on inflection that requires the entire sentence,
the latter on the translator's freedom to reorder components),
1:1 fragment translation breaks in different ways for different languages:
- **SOV (ja / ko / tr)**:
Verb-final word order means splits like
`"These settings are"` + `added` + `"to any existing..."`
cannot be put into natural order when translated fragment-by-fragment
- **Germanic (de)**:
Gender/case agreement between article and noun becomes hard to maintain when they sit in separate fragments
- **Romance (es / fr / it / pt_BR)**:
Adjective placement, mood (subjunctive / imperative), verb number agreement,
and other entire-sentence-dependent elements don't fit cleanly at fragment boundaries
- **Slavic (cs / uk)**:
Case (determined by preposition/verb relationships) is easy to get wrong when fragmented
- **Chinese (zh / zh_Hant_TW)**:
Modifier position and other SVO-but-different-from-English adjustments
cannot be applied at the fragment level
#### Workaround I introduced in earlier PRs #4019 / #4050
As a countermeasure for this word-order difference, in my earlier PRs #4019 / #4050
I introduced **redistribution of translations across adjacent fragments** to get natural word order.
When a fragment becomes unnecessary in SOV languages as a result of redistribution,
it is set to `msgstr " "` (a single space) to hide it
(because `msgstr ""` would fall back to the English source).
#### Side effect of the workaround
As soon as a short-worded `msgid` is reused in a different template,
the translation redistributed for the original context gets applied verbatim and breaks the meaning.
**Concrete example**:
`"Set to"` currently has `msgstr " "`, so any new template using `"Set to"` would render as blank in Japanese.
As I already noted in PR #4050's Side Note,
"consolidating these fragmented `msgid`s into a single one would greatly simplify maintenance."
### Problem 2: `` italics that degrade legibility in CJK languages are forced from the template side
CJK fonts often have no italic cut, so `` falls back to a mechanical slant that reduces legibility.
In the current fragmented structure where `` sits outside `_()`,
CJK translators — looking only at the `msgid` — cannot notice `` exists,
and the template HTML forcibly applies italics.
Embedding `` inside the `msgid` lets translators decide per language
whether to "keep / drop / replace with `` / wrap in quotation marks".
## Consolidation pattern aligned with the GNU gettext manual
We follow the guidelines from the GNU gettext manual's [Preparing Translatable Strings](https://www.gnu.org/software/gettext/manual/html_node/Preparing-Strings.html):
> [!IMPORTANT]
> - [Entire sentences](https://www.gnu.org/software/gettext/manual/html_node/Entire-sentences.html):
> Translatable strings should be entire sentences.
> Because gender/number declension depends on other parts of the sentence,
> half-sentence *"dumb string concatenation"* breaks in many languages other than English.
> - [No string concatenation](https://www.gnu.org/software/gettext/manual/html_node/No-string-concatenation.html):
> Placing adjacent `_()` calls is semantically equivalent to runtime `strcat` concatenation,
> so the same guideline applies.
> The manual also notes that
> "in some languages the translator might want to swap the order" of components.
> - [No embedded URLs](https://www.gnu.org/software/gettext/manual/html_node/No-embedded-URLs.html):
> URLs should not be written directly inside `msgid`s;
> they should be injected via `%(name)s` placeholders,
> and values passed **as kwargs**
> (*"Use placeholders in format strings instead of embedded URLs."*).
> This proposal extends the same principle to other dynamic values —
> Jinja variables, literal `{{}}` escapes, and `{% if %}` conditional branches.
> - [No unusual markup](https://www.gnu.org/software/gettext/manual/html_node/No-unusual-markup.html):
> Embedding `` / `` / `` HTML inside `msgid`s is acceptable
> (*"HTML markup, however, is common enough that it's probably ok to use in translatable strings."*).
### Key points of the policy
The approach common to the patterns below:
1. **Do not fragment msgids**:
keep each translatable unit at the sentence (or paragraph) level, never split mid-way for dynamic values or markup
2. **Inject all dynamic values via placeholders**:
regardless of kind — URL, Jinja variable, literal `{{}}` escape —
inject them from outside via `%(name)s` + kwargs, and keep them out of the msgid structure
3. **Inline markup inside the msgid**:
`` / `` / `` count as word-level decoration — inside the msgid, rendered with `|safe`.
### Post-Consolidation patterns
#### Pattern 1: Inline HTML embedding
```jinja
{# Before: split into 3 fragments. sits outside _(), so CJK translators
cannot see it from the msgid and the mechanical italic slant degrades legibility #}
{{ _('Helps reduce changes detected caused by sites shuffling lines around, combine with') }} {{ _('check unique lines') }} {{ _('below.') }}
{# After: 1 msgid with inline , rendered with |safe.
CJK translators can now decide per language to keep / drop / replace with / wrap in quotation marks
(e.g., both ja and zh in this repo chose to drop and wrap the term in quotation marks) #}
{{ _('Helps reduce changes detected caused by sites shuffling lines around, combine with check unique lines below.') | safe }}
```
#### Pattern 2: URL as kwarg
```jinja
{# Before: `"Use"` + literal `` + `"AppRise Notification URLs"` + `` + `"for notification to just about any service!"` — 3 fragments #}
{{ _('Use') }} {{ _('AppRise Notification URLs') }} {{ _('for notification to just about any service!') }}
{# After: URL passed as kwarg, embedded inside the msgid #}
{{ _('Use AppRise Notification URLs for notification to just about any service!',
url='https://github.com/caronc/apprise') | safe }}
```
#### Pattern 3: Literal `{{}}` escape as kwarg
```jinja
{# Before: literal {{token}} in the middle forces fragment splitting.
Writing {{}} inside the msgid would be double-interpolated by Jinja, so it's kept outside. #}
{{ _('Accepts the') }} {{ '{{token}}' }} {{ _('placeholders listed below') }}
{# After: literal passed as kwarg; the msgid stays as an entire sentence with just %(token)s #}
{{ _('Accepts the %(token)s placeholders listed below', token='{{token}}') | safe }}
```
#### Pattern 4: `{% if %}` outside msgid
```jinja
{# Before: {% if %} adds a fragment inside the msgid.
Split into `"URL or Title"` + `"in"` + literal `'{{ active_tag.title }}'` (3 fragments) #}
{{ _('URL or Title') }}{% if active_tag_uuid %} {{ _('in') }} '{{ active_tag.title }}'{% endif %}
{# After: {% if %} moved outside the msgid; branch between complete msgids.
Each language can freely rearrange the %(title)s position
(e.g., ja「'X' 内の URL またはタイトル」, zh「'X' 中的 URL 或标题」, tr「'X' içinde URL veya Başlık」) #}
{% if active_tag_uuid %}
{{ _("URL or Title in '%(title)s'", title=active_tag.title) }}
{% else %}
{{ _('URL or Title') }}
{% endif %}
```
## How to update locales
1. Consolidate templates, then regenerate `messages.pot` with `python setup.py extract_messages`
2. Propagate new `msgid`s to each `.po` with `python setup.py update_catalog`
3. Locale translation policy:
- Refer to existing fragment translations and reassemble them into a natural entire sentence
- When existing fragment translations are **incomplete (missing, etc.)**,
leave `msgstr ""` so English fallback takes effect
- When existing fragment translations are **grammatically incorrect or unnatural**,
fix them into natural translations (including per-language word-order adjustments;
tools like Claude Code Opus with 1M context may be used)
4. **Unified CJK `` handling**:
For `ja` / `ko`/ `zh` / `zh_Hant_TW`, apply "`` replacement / dropping / wrapping in quotation marks"
Feedback on this approach is welcome.
Contributor guide
Research direction
Start by locating the Jinja templates containing fragmented gettext calls and review the existing messages.pot and locale .po files. Run `python setup.py extract_messages` and `python setup.py update_catalog` after consolidating sentence-level msgids and injecting dynamic values as kwargs. Done means the affected templates use complete translatable units and the locale catalogs preserve or intentionally update their translations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- internationalization, localization
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100