deepchecks / deepchecks/deepchecks

Text Data Duplicates

Open
#2,387 0 comments 0 reactions 0 assignees View on GitHub
feature linear nlp
Dominant language
Python
Stars
4.1k
Forks
303
PR merge metrics
No merged PRs in 30d

Description

Find data duplicates in text (not similarity, which is covered in #2385 )

Can do 3 checks:
* find data duplicates in 1 dataset
* find data duplicates in 2 datasets
* find data duplicates in 1 dataset with conflicting labels

Contributor guide

Open the contributing guide

Research direction

Start by reading issue #2385 to understand the existing similarity checks, then inspect the repository's data-validation and text-data entry points. Define how exact duplicates are detected within one dataset, across two datasets, and when identical text has conflicting labels. Done means all three checks are implemented with clear results and coverage for each case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.