WeblateOrg / WeblateOrg/weblate

Weblate fills in syntactically wrong `Language:` field in PO header for languages with script

Open
#18,159 3 comments 0 reactions 0 assignees View on GitHub
Blocked by upstream Waiting for: Demand
Dominant language
Python
Stars
6.1k
Forks
1.4k
Avg merge
9h 53m
Merged PRs (30d)
395

Description

### Describe the issue

If you create a new language with a script, like "Chinese (Simplified Han Script)" (`zh_Hans`) and let Weblate generate the PO files from it, then Weblate will fill in a syntactically wrong value in the PO file's `Language:` field in the header.

It will fill in `zh_Hans`. This is syntactically invalid.

The [manual](https://www.gnu.org/software/gettext/manual/gettext.html#Header-Entry) for gettext 1.0 allows only 3 syntaxes:

1) ``
2) `_`
3) `_@`

Where `` is a 2-letter language code, `` is a 2- or 3 letter language code, `` is a 2-letter country code and `` is a variant, which can be any lowercase string and may be used to specify the script. The manual does not define any further restrictions for ``.

`zh_Hans` does not fit any of the valid syntaxes and is thus a syntax error.

This is not just a theoretical problem. I get an error message from Poedit when I try to open a PO file with `zh_Hans` specified in `Language`, likely due to the non-standard syntax. It's possible other applications that read the `Language:` field will get confused as well.

### I already tried

- [x] I've read and searched [the documentation](https://docs.weblate.org/).
- [x] I've searched for similar filed issues in this repository.

### Steps to reproduce the behavior

1. Make sure you have a project with gettext PO files used as the file format
2. In a project, click on the "Languages" tab
3. Click on the "+" to add a new language
4. Add a language with a script specified
5. Use the Weblate interface to download the PO file for that language
6. Check the `Language:` field in the PO file in a text editor

### Expected behavior

Weblate guarantees to always insert a syntactically valid string in the `Language` field.

#### Solution 1

The conversion to PO files could use this generalized algorithm for languages with script:

1. Start with the ISO 639 language code
2. If an ISO 3166 country code is specified, append `_`, otherwise append `_ZZ`
3. Append `@` where `<script>` is the ISO-15924 script code converted to lowercase

Notes:
* The country code is necessary because that's the only syntactically valid way to add a variant designator
* The `ZZ` is a custom user-assignable ISO country code which I use as a dummy that does not make any claim about any nation. The choice of `ZZ` is arbitrary, I could have picked any other free user-assignable code
* According to the [FAQ](https://www.unicode.org/iso15924/faq.html#17), ISO 15924 script codes are case-insensitive

Examples of the codes the algorithm would generate (input left, output right):

* Chinese (Simplified Han Script): `zh_Hans` → `zh_ZZ@hans`
* Chinese (Traditional Han Script): `zh_Hant` → `zh_ZZ@hant`
* Belarusian (Latin script): `be_Latn` → `be_ZZ@latn`
* Serbian (Cyrillic Script): `sr_Cryl` → `sr_ZZ@cyrl`
* Chinese (Traditional Han Script, Taiwan): `zh_Hant_TW` → `zh_TW@hant`

#### Solution 2

When writing the Language field, just put the language code (and country, if it exists) but drop the script entirely. Thus:

* `zh_Hant` → `zh`
* `zh_Hans` → `zh`
* `zh_Hant_TW` → `zh_TW`

But the obvious downside is that you're throwing away information.

### How do you run Weblate?

I am an user on translate.codeberg.org.

### Weblate versions

5.15.2

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.