huggingface / huggingface/datatrove

Naming Gopher's "max_non_alpha_words_ratio"

Open
#289 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

In the Gopher filter, there's this filter

```
# that 80 % of words in a document contain at least one alphabetic character
if (
self.max_non_alpha_words_ratio
and sum([any((c.isalpha() for c in w)) for w in words]) / n_words < self.max_non_alpha_words_ratio
):
return False, "gopher_below_alpha_threshold"
```

Given that all documents that have a LOWER ratio are removed, I would expect the variable name to be _min_non_alpha_words_ratio_, similar to all other variable names.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.