bigscience-workshop / bigscience-workshop/catalogue_data
Wiki-based dataset cleaning
Open
- Dominant language
- Jupyter Notebook
- Stars
- 8
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Wiki-based dataset that are not wikipedia have at least the following issues:
- template text being repeated
- non-article pages (users, categories)
- some wikipedia formatting noise (may be only in the non-article pages though)
Some solutions already mentioned to deal with those:
- Looking at the `type` field in `meta`: keeping only `text` types seems very strong
- Deduplicating to remove templates
- Looking at the `title` field in meta: allows to remove user pages for example
@SaulLu @cakiki
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.