Pipeline for data contamination

Hi guys,

How can I implement a pipeline to check for contamination between two different datasets (e.g., pre-training vs. fine-tuning datasets) and eventually delete marked documents from the pre-training dataset?

Thanks.

No files, tests, or entry points are named. Start by reviewing datatrove's customizable pipeline processing blocks and dataset handling, then clarify how contamination is detected, how documents are marked, and whether deletion is required. Done should include an agreed pipeline design and verified behavior for identifying and removing contaminated pre-training documents.

Written by the indexing model from the issue text.