google / google/langextract

Russian text Unicode error

Open
#139 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
38.6k
Forks
2.7k
Avg merge
3d 15h
Merged PRs (30d)
3

Description

Great work, but I encountered an error when extracting entities from mixed Russian and English text. The core error line is as follows:
File "D:\project\07-multilan\langextract_example.py", line 43, in
lx.io.save_annotated_documents([result], output_name="extraction_results.jsonl")
File "D:\project\07-multilan\.venv\lib\site-packages\langextract\io.py", line 123, in save_annotated_documents
f.write(json.dumps(doc_dict, ensure_ascii=False) + '\n')
UnicodeEncodeError: 'gbk' codec can't encode character '\u0301' in position 12984: illegal multibyte sequence

Contributor guide

Open the contributing guide

Research direction

Start with langextract/io.py around line 123 and reproduce the failure using langextract_example.py with mixed Russian and English text on the reported environment. Trace how extraction_results.jsonl is written and verify that saving the annotated document completes without a UnicodeEncodeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
internationalization
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.