WeblateOrg / WeblateOrg/weblate
Correct source translations are required for CSV imports even in monolingual setup
- Dominant language
- Python
- Stars
- 6.1k
- Forks
- 1.4k
- Avg merge
- 9h 53m
- Merged PRs (30d)
- 395
Description
### Describe the issue
GIVEN
- a weblate instance with
- one project,
- one component configured to use
- JSON file format internally
- `en_US` as the source language
- `en_US.json` as the monolingual base language file with base file edit enabled
- two languages in use; `en_US` and `fr`
- some strings present in the base language `en_US`, but not in `fr`
WHEN
- attempting to provide translations of those strings using a CSV file with the columns `source`, `target` and `context`, where
- the `source` column either does not exist at all, contains empty strings or a string that is outdated, i.e., the current `en_US` translation is different from the string provided for the same key
- the `target` column contains the `fr` translations to be imported, and
- the `context` column contains the keys
THEN
- importing is not possible and leads to a `not found` report after uploading the CSV file in `Add as translation` mode
### I already tried
- [X] I've read and searched [the documentation](https://docs.weblate.org/).
- [X] I've searched for similar issues in this repository.
### Steps to reproduce the behavior
1. Setup a Weblate project and component as described above
2. Import the following CSV file for `en_US`:
```
"location","source","target","id","fuzzy","context","translator_comments","developer_comments"
"","Old Translation","Old Translation","","","key","",""
```
3. Download a CSV file for `fr`, which should look like this:
```
"location","source","target","id","fuzzy","context","translator_comments","developer_comments"
"","Old Translation","","","False","key","",""
```
4. Update the `en_US` string to something else, e.g. "New Translation", using another upload or the GUI
5. Add a translation to the CSV file downloaded in step 3 and upload it:
```
"location","source","target","id","fuzzy","context","translator_comments","developer_comments"
"","Old Translation","Traduction en français","","False","key","",""
```
6. Observe that the upload of the new `fr` string fails:
```
Processed 1 string from the uploaded files (skipped: 0, not found: 1, updated: 0).
```
7. Remove the source translation from the CSV file in step 5 and upload it again:
```
"location","source","target","id","fuzzy","context","translator_comments","developer_comments"
"","","Traduction en français","","False","key","",""
```
8. Observe that once again the upload of the new `fr` string fails:
```
Processed 1 string from the uploaded files (skipped: 0, not found: 1, updated: 0).
```
9. Remove the entire source column from the CSV file in step 7 and upload it again:
```
"location","target","id","fuzzy","context","translator_comments","developer_comments"
"","Traduction en français","","False","key","",""
```
10. Observe that yet again the upload of the new `fr` string fails:
```
Processed 1 string from the uploaded files (skipped: 0, not found: 1, updated: 0).
```
### Expected behavior
Since we are working with a monolingual base language file, whose functionality is described in the documentation using the words
> monolingual formats identify the string by ID, and each language file contains only the mapping of those to any given language
I expect the import to succeed in both step 6 and 8 since the correct key was specified in both files, and the content of the `source` column should not matter (the `source` column should not even be required).
For a monolingual setup as described here, I expect that the key is used for identifying a string rather than its translation in the base language. That’s the whole point in using a monolingual format.
### Screenshots
_No response_
### Exception traceback
_No response_
### How do you run Weblate?
Docker container
### Weblate versions
* Weblate: 4.14.2
* Django: 4.1.3
* siphashc: 2.1
* translate-toolkit: 3.7.4
* lxml: 4.9.1
* Pillow: 9.3.0
* bleach: 5.0.1
* python-dateutil: 2.8.2
* social-auth-core: 4.3.0
* social-auth-app-django: 5.0.0
* django-crispy-forms: 1.14.0
* oauthlib: 3.2.2
* django-compressor: 4.1
* djangorestframework: 3.14.0
* django-filter: 22.1
* django-appconf: 1.0.5
* user-agents: 2.2.0
* filelock: 3.8.0
* rapidfuzz: 2.13.1
* openpyxl: 3.0.10
* celery: 5.2.7
* kombu: 5.2.4
* translation-finder: 2.14
* weblate-language-data: 2022.8
* html2text: 2020.1.16
* pycairo: 1.21.0
* pygobject: 3.42.2
* diff-match-patch: 20200713
* requests: 2.28.1
* django-redis: 5.2.0
* hiredis: 2.0.0
* sentry_sdk: 1.10.1
* Cython: 0.29.32
* misaka: 2.1.1
* GitPython: 3.1.29
* borgbackup: 1.2.2
* pyparsing: 3.0.9
* pyahocorasick: 1.4.4
* python-redis-lock: 4.0.0
* charset-normalizer: 2.1.1
* Python: 3.11.0
* Git: 2.30.2
* psycopg2: 2.9.5
* psycopg2-binary: 2.9.5
* phply: 1.2.5
* ruamel.yaml: 0.17.21
* tesserocr: 2.5.2
* boto3: 1.26.3
* zeep: 4.2.0
* aeidon: 1.11
* iniparse: 0.5
* mysqlclient: 2.1.1
* Mercurial: 6.2.3
* git-svn: 2.30.2
* git-review: 2.3.1
* Redis server: 6.2.7
* PostgreSQL server: 15.1
* Database backends: django.db.backends.postgresql
* Cache backends: default:RedisCache, avatar:FileBasedCache
* Email setup: django.core.mail.backends.smtp.EmailBackend: 127.0.0.1
* OS encoding: filesystem=utf-8, default=utf-8
* Celery: redis://cache:6379/1, redis://cache:6379/1, regular
* Platform: Linux 5.15.49-linuxkit (aarch64)
### Weblate deploy checks
```shell
(MWE setup to reproduce the issue, so configuration is not used in production. Results shown below should not be relevant to the issue.)
SystemCheckError: System check identified some issues:
CRITICALS:
?: (weblate.E003) Cannot send e-mail ([Errno 111] Connection refused), please check EMAIL_* settings.
HINT: https://docs.weblate.org/en/weblate-4.14.2/admin/install.html#out-mail
?: (weblate.E012) The server e-mail address should be changed from its default value
HINT: https://docs.weblate.org/en/weblate-4.14.2/admin/install.html#production-email
?: (weblate.E013) The "From" e-mail address should be changed from its default value
HINT: https://docs.weblate.org/en/weblate-4.14.2/admin/install.html#production-email
WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W018) You should not have DEBUG set to True in deployment.
INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
HINT: https://docs.weblate.org/en/weblate-4.14.2/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
HINT: https://docs.weblate.org/en/weblate-4.14.2/admin/backup.html
System check identified 9 issues (1 silenced).
```
### Additional context
This problem occurred in our project’s workflow in which
1. a developer introduces a large amount of strings and imports both the keys and a preliminary `en_US` translation for them
2. a CSV file is downloaded for several languages, including the `en_US` base language, and sent to language experts to (for `en_US`) improve and update the preliminary translations or (for all other languages) to provide translations for those strings
3. we receive the file with updated base language translations from the `en_US` expert and import them
4. we receive translation files for other languages containing the now outdated base language translations as `source`, leading to the described `not found` import failure
Contributor guide
Assessment
This issue has not been assessed yet.