huggingface / huggingface/datatrove

Get list of duplicate docs id's from minhash dedup.

Open
#213 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

As I understand, minhash dedup, creates `remove` files for documents to remove, as well as the option to create a `clusters` file which includes duplicate document and which cluster they belong to.

Maybe a duplicate of #209, but is there a way to get a list of all documents which have duplicates (not only documents to remove)? I suspect the `clusters` file might have that info but on inspection I see some clusters with only 1 document, which is also on the list of documents to remove `remove`. Is it the case that it's duplicate pair not included in the `clusters` file?

eg I see something like this:
```
.clusters

doc id cluster_id
doc0 0
doc1 0
doc2 0
doc3 1

.remove

doc1
doc2
doc3

# I'm curious on the duplicate pair for doc3
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the minhash dedup implementation and reading how the `remove` and `clusters` files are generated. Reproduce the example with a duplicate pair, then determine whether the clusters output includes all duplicate documents. Done means the expected relationship between both outputs is documented or the requested complete duplicate listing is supported and tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.