deepchecks / deepchecks/deepchecks
Text Data Duplicates
Open
feature
linear
nlp
- Dominant language
- Python
- Stars
- 4.1k
- Forks
- 303
- PR merge metrics
- No merged PRs in 30d
Description
Find data duplicates in text (not similarity, which is covered in #2385 )
Can do 3 checks:
* find data duplicates in 1 dataset
* find data duplicates in 2 datasets
* find data duplicates in 1 dataset with conflicting labels
Contributor guide
Research direction
Start by reading issue #2385 to understand the existing similarity checks, then inspect the repository's data-validation and text-data entry points. Define how exact duplicates are detected within one dataset, across two datasets, and when identical text has conflicting labels. Done means all three checks are implemented with clear results and coverage for each case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100