WikiExtractor / WikiExtractor/wikiextractor
handling history
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4k
- Forks
- 1k
- Avg merge
- 1m
- Merged PRs (30d)
- 10
Description
In cases where a has multiple tags (because history is included in the dump) it seems as if the extractor will generate one long file that concatenates the history into one long document instead of breaking it out into multiple files (with multiple revision tags). I would expect wikiextractor to break the history into individual revisions when I use the -r argument.
To replicate, go to https://en.wikipedia.org/wiki/Special:Export, add any article in the box that says "Add pages manually" and uncheck the box that says "Include only the current revision, not the full history." If you down load the special export and then run Wikiextractor with the -r argument and it will generate the long, concatenated file.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with a Wikipedia Special:Export dump containing full history, then run Wikiextractor with the -r argument and inspect how multiple tags are handled. Done means each revision is extracted into its own file rather than concatenated into one document.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100