openedx / openedx/i18n-tools

UnicodeDecodeError while logging

Open
#79 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
26
Forks
36
PR merge metrics
No merged PRs in 30d

Description

Running the i18n_tool validate command kept failing because of some UnicodeDecodeError that occurred when logging the problem with the .po file

Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/bin/i18n_tool", line 11, in <module>
    sys.exit(main())
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/main.py", line 60, in main
    return module.main()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/__init__.py", line 51, in __call__
    return self.run(args)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/validate.py", line 250, in run
    if validate_po_files(self.configuration, locale_dir, report_empty=args.empty, check_all=args.check_all):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/validate.py", line 43, in validate_po_files
    if msgfmt_check_po_file(locale_dir, filename):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/validate.py", line 80, in msgfmt_check_po_file
    log.warning(u'\n' + err.decode('utf8'))
  File "/edx/app/edxapp/venvs/edxapp/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 328: invalid continuation byte

This was the error(err) that it was trying to log in log.warning(u'\n' + err.decode('utf8'))

../../Desktop/Project/i18n-tools/i18n/django-partial.po:6509: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, there is an unterminated format directive.\n../../Desktop/Project/i18n-tools/i18n/django-partial.po:9602: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '\xd0' cannot start a field name.\n../../Desktop/Project/i18n-tools/i18n/django-partial.po:12407: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '\xd0' cannot start a field name.\nmsgfmt: found 3 fatal errors\n"
663
../../Desktop/Project/i18n-tools/i18n/django-partial.po:6509: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, there is an unterminated format directive.
../../Desktop/Project/i18n-tools/i18n/django-partial.po:9602: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '�' cannot start a field name.
../../Desktop/Project/i18n-tools/i18n/django-partial.po:12407: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '�' cannot start a field name.
msgfmt: found 3 fatal errors

Since the filename or language code was not getting logged for which the validation was failing on jenkins pull_translations job it was particularly difficult to find the file and issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in i18n/validate.py at msgfmt_check_po_file, especially the log.warning call around line 80, and reproduce the failure with i18n_tool validate on the reported .po file. Check that validation errors can be logged without a UnicodeDecodeError and that the failing filename or language code is identifiable in the output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, localization
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.