WeblateOrg / WeblateOrg/weblate
Weblate fills in syntactically wrong `Language:` field in PO header for languages with script
- Dominant language
- Python
- Stars
- 6.1k
- Forks
- 1.4k
- Avg merge
- 9h 53m
- Merged PRs (30d)
- 395
Description
### Describe the issue
If you create a new language with a script, like "Chinese (Simplified Han Script)" (`zh_Hans`) and let Weblate generate the PO files from it, then Weblate will fill in a syntactically wrong value in the PO file's `Language:` field in the header.
It will fill in `zh_Hans`. This is syntactically invalid.
The [manual](https://www.gnu.org/software/gettext/manual/gettext.html#Header-Entry) for gettext 1.0 allows only 3 syntaxes:
1) ``
2) `_`
3) `_@`
Where `` is a 2-letter language code, `` is a 2- or 3 letter language code, `` is a 2-letter country code and `` is a variant, which can be any lowercase string and may be used to specify the script. The manual does not define any further restrictions for ``.
`zh_Hans` does not fit any of the valid syntaxes and is thus a syntax error.
This is not just a theoretical problem. I get an error message from Poedit when I try to open a PO file with `zh_Hans` specified in `Language`, likely due to the non-standard syntax. It's possible other applications that read the `Language:` field will get confused as well.
### I already tried
- [x] I've read and searched [the documentation](https://docs.weblate.org/).
- [x] I've searched for similar filed issues in this repository.
### Steps to reproduce the behavior
1. Make sure you have a project with gettext PO files used as the file format
2. In a project, click on the "Languages" tab
3. Click on the "+" to add a new language
4. Add a language with a script specified
5. Use the Weblate interface to download the PO file for that language
6. Check the `Language:` field in the PO file in a text editor
### Expected behavior
Weblate guarantees to always insert a syntactically valid string in the `Language` field.
#### Solution 1
The conversion to PO files could use this generalized algorithm for languages with script:
1. Start with the ISO 639 language code
2. If an ISO 3166 country code is specified, append `_`, otherwise append `_ZZ`
3. Append `@` where `<script>` is the ISO-15924 script code converted to lowercase
Notes:
* The country code is necessary because that's the only syntactically valid way to add a variant designator
* The `ZZ` is a custom user-assignable ISO country code which I use as a dummy that does not make any claim about any nation. The choice of `ZZ` is arbitrary, I could have picked any other free user-assignable code
* According to the [FAQ](https://www.unicode.org/iso15924/faq.html#17), ISO 15924 script codes are case-insensitive
Examples of the codes the algorithm would generate (input left, output right):
* Chinese (Simplified Han Script): `zh_Hans` → `zh_ZZ@hans`
* Chinese (Traditional Han Script): `zh_Hant` → `zh_ZZ@hant`
* Belarusian (Latin script): `be_Latn` → `be_ZZ@latn`
* Serbian (Cyrillic Script): `sr_Cryl` → `sr_ZZ@cyrl`
* Chinese (Traditional Han Script, Taiwan): `zh_Hant_TW` → `zh_TW@hant`
#### Solution 2
When writing the Language field, just put the language code (and country, if it exists) but drop the script entirely. Thus:
* `zh_Hant` → `zh`
* `zh_Hans` → `zh`
* `zh_Hant_TW` → `zh_TW`
But the obvious downside is that you're throwing away information.
### How do you run Weblate?
I am an user on translate.codeberg.org.
### Weblate versions
5.15.2
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.