bigscience-workshop / bigscience-workshop/data-preparation

Why stopwords_min_cutoff rather than stopwords_max_cutoff?

Open
#43 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
318
Forks
42
PR merge metrics
No merged PRs in 30d

Description

Thanks for your helpful codebase!

I am a bit confused about `stop words filtering`.
The release code removes the document, if its stop words ratio below the certain cutoff.
https://github.com/bigscience-workshop/data-preparation/blob/9d0588419073cc5bf0fb92b58f37f2a1016572c3/preprocessing/training/01b_oscar_cleaning_and_filtering/filtering.py#L590
But in [notebook](https://drive.google.com/file/d/1cCJ8sWE88TRLDAa3eHLmXO4JlkR2QzLY/view), section 2.5 states `If the stop words ratio for a document is higher than a certain cutoff, it is removed.`

I am wondering which one is more useful in your practice.
Thanks in advance!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.