WikiExtractor / WikiExtractor/wikiextractor
UnicodeDecodeError when extracting from an (decompressed) .xml dump
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4k
- Forks
- 1k
- Avg merge
- 1m
- Merged PRs (30d)
- 10
Description
Hi,
I am getting UnicodeDecodeErrors when I try to extract a decompressed .xml dump. For the record, this is how I am using the WikiExtractor:
WikiExtractor.py wikicorpus_en.xml -b 100M --processes 50 -o /path/to/extraction/folder/
I am suspecting this line to be at the origin of the problem since fileinput.hook_compressed seems to result in opening the file using open(filename, mode) instead of open(filename, mode, encoding='utf-8') which would avoid the decoding error.
Changing the line with this one solved the issue for me:
input = fileinput.FileInput(input_file, openhook=fileinput.hook_encoded(encoding='utf-8'))
If the tool wasn't intended to be used with an already decompressed .xml dump you can close this issue. Thank you.
Hicham
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at WikiExtractor.py around line 2871 and reproduce the reported command with an already decompressed wikicorpus_en.xml dump. Check the input-opening path and verify that extraction completes without UnicodeDecodeError when the XML is UTF-8 encoded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100