UnicodeDecodeError while logging
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 36
- PR merge metrics
- No merged PRs in 30d
Description
Running the i18n_tool validate command kept failing because of some UnicodeDecodeError that occurred when logging the problem with the .po file
Traceback (most recent call last):
File "/edx/app/edxapp/venvs/edxapp/bin/i18n_tool", line 11, in <module>
sys.exit(main())
File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/main.py", line 60, in main
return module.main()
File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/__init__.py", line 51, in __call__
return self.run(args)
File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/validate.py", line 250, in run
if validate_po_files(self.configuration, locale_dir, report_empty=args.empty, check_all=args.check_all):
File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/validate.py", line 43, in validate_po_files
if msgfmt_check_po_file(locale_dir, filename):
File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/validate.py", line 80, in msgfmt_check_po_file
log.warning(u'\n' + err.decode('utf8'))
File "/edx/app/edxapp/venvs/edxapp/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 328: invalid continuation byte
This was the error(err) that it was trying to log in log.warning(u'\n' + err.decode('utf8'))
../../Desktop/Project/i18n-tools/i18n/django-partial.po:6509: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, there is an unterminated format directive.\n../../Desktop/Project/i18n-tools/i18n/django-partial.po:9602: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '\xd0' cannot start a field name.\n../../Desktop/Project/i18n-tools/i18n/django-partial.po:12407: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '\xd0' cannot start a field name.\nmsgfmt: found 3 fatal errors\n"
663
../../Desktop/Project/i18n-tools/i18n/django-partial.po:6509: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, there is an unterminated format directive.
../../Desktop/Project/i18n-tools/i18n/django-partial.po:9602: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '�' cannot start a field name.
../../Desktop/Project/i18n-tools/i18n/django-partial.po:12407: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '�' cannot start a field name.
msgfmt: found 3 fatal errors
Since the filename or language code was not getting logged for which the validation was failing on jenkins pull_translations job it was particularly difficult to find the file and issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in i18n/validate.py at msgfmt_check_po_file, especially the log.warning call around line 80, and reproduce the failure with i18n_tool validate on the reported .po file. Check that validation errors can be logged without a UnicodeDecodeError and that the failing filename or language code is identifiable in the output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, localization
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100