bigscience-workshop / bigscience-workshop/data-preparation

Changing parmater values to extreme in parameters_filtering.py doesn't change the no. f documents being removed

Open
#42 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
318
Forks
42
PR merge metrics
No merged PRs in 30d

Description

Hi! Kudos to the author for an end-to-end piepline for cleaning and filtering a large corpus. I was working with [main_filtering.py](https://github.com/bigscience-workshop/data-preparation/blob/main/preprocessing/training/01b_oscar_cleaning_and_filtering/main_filtering.py) and was trying to change the parameter values in parameters_filtering.py, hoping to increase/decrease the no. of documents that were being removed out. But I observe no changes.

1. I have english dataset so I set parameters_filtering_en, and I have experimented with the given values and some modifications in 1/more conditions and cutoffs.
2. I have also tried out parameters_filtering_default where I do observe changes in documents being filtered out. The no. was different from those in parameters_filtering_en.

- The parameters_filtering_default has some error. I modified languages_id.py to account for "defualt" as langauge but used flagged_/stop_words of english language.

3. Within parameters_filtering_default or parameters_filtering_en, when parameter values are changed no changes are observed in no. of documents or documents which are getting removed.

Kindly review the code and let me know the solutions. Also let me know if I'm missing something.

Thank You!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.