huggingface / huggingface/datatrove
Naming Gopher's "max_non_alpha_words_ratio"
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
In the Gopher filter, there's this filter
```
# that 80 % of words in a document contain at least one alphabetic character
if (
self.max_non_alpha_words_ratio
and sum([any((c.isalpha() for c in w)) for w in words]) / n_words < self.max_non_alpha_words_ratio
):
return False, "gopher_below_alpha_threshold"
```
Given that all documents that have a LOWER ratio are removed, I would expect the variable name to be _min_non_alpha_words_ratio_, similar to all other variable names.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.