bigscience-workshop / bigscience-workshop/catalogue_data
Catching crawling noise + ads
Open
- Dominant language
- Jupyter Notebook
- Stars
- 8
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Some datasets (f.e. `bigscience-catalogue-lm-data/lm_es_pseudocrawl-filtered_396_www_eldiario_es`) happen to have a mix of:
- crawling noise
- ad javascript
that results in code-like snippets that can be caught by looking for `{` and `}`s. I have seen some amount of typo `{` that are unrelated for some reason, I don't think we want to remove them but I don't think we really care either. I'd advocate either removing any line in the pseudocrawled newspaper datasets that contain `}`, or removing all of the `{...}` groups.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.