huggingface / huggingface/datatrove
Explanation of what exactly the various dedup strategies do
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Is there any documentation of what the various deduplication strategies actually do? My use case is that I have a corpus of texts and the document have certain template texts that reoccur with slight variations to them. I don't know if any of the strategies here fit for that, and the code examples and whatever documentation there is don't help. For instance what does "only_dedup_in_index" mean for SentenceDedup?
Maybe it's just because I'm not very familiar with deduplication in general that I'm struggling with this, so would appreciate help.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.