huggingface / huggingface/datatrove
assert in url dedup
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
https://github.com/huggingface/datatrove/blob/b9fb72a21350fd9ed13a115a6d24ea5f58015ae7/src/datatrove/pipeline/dedup/url_dedup.py#L103-L106
I do not understand why we cannot use 0 as priority. Actually in the function provided by the example script, some doc will have a priority of 0 if len < 4.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.