Russian text Unicode error
- Dominant language
- Python
- Stars
- 38.6k
- Forks
- 2.7k
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 3
Description
Great work, but I encountered an error when extracting entities from mixed Russian and English text. The core error line is as follows:
File "D:\project\07-multilan\langextract_example.py", line 43, in
lx.io.save_annotated_documents([result], output_name="extraction_results.jsonl")
File "D:\project\07-multilan\.venv\lib\site-packages\langextract\io.py", line 123, in save_annotated_documents
f.write(json.dumps(doc_dict, ensure_ascii=False) + '\n')
UnicodeEncodeError: 'gbk' codec can't encode character '\u0301' in position 12984: illegal multibyte sequence
Contributor guide
Research direction
Start with langextract/io.py around line 123 and reproduce the failure using langextract_example.py with mixed Russian and English text on the reported environment. Trace how extraction_results.jsonl is written and verify that saving the annotated document completes without a UnicodeEncodeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- internationalization
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100