How do I assess whether the accuracy before and after dedup

https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py
After I finish executing this file, I have two folders: **deduplicated_output** and **removed**. How do I determine which of the removed samples corresponds to which of the retained samples

Start by reading examples/fineweb.py and inspecting how it produces the deduplicated_output and removed folders. Determine whether the example exposes enough information to relate removed samples to retained samples, then document the steps and limitations for assessing the before-and-after accuracy.

Written by the indexing model from the issue text.