huggingface / huggingface/datatrove
Get list of duplicate docs id's from minhash dedup.
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
As I understand, minhash dedup, creates `remove` files for documents to remove, as well as the option to create a `clusters` file which includes duplicate document and which cluster they belong to.
Maybe a duplicate of #209, but is there a way to get a list of all documents which have duplicates (not only documents to remove)? I suspect the `clusters` file might have that info but on inspection I see some clusters with only 1 document, which is also on the list of documents to remove `remove`. Is it the case that it's duplicate pair not included in the `clusters` file?
eg I see something like this:
```
.clusters
doc id cluster_id
doc0 0
doc1 0
doc2 0
doc3 1
.remove
doc1
doc2
doc3
# I'm curious on the duplicate pair for doc3
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the minhash dedup implementation and reading how the `remove` and `clusters` files are generated. Reproduce the example with a duplicate pair, then determine whether the clusters output includes all duplicate documents. Done means the expected relationship between both outputs is documented or the requested complete duplicate listing is supported and tested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100