Feture request: Data deduplication and Near-deduplication

In current Microsoft.ML developers may need to reduce size of huge datasets (#6679) or at least it might be advisable to do so: For many problems and algorithms hyperparameters tuning is important and may improve results more than increasing size of dataset. Lower datasets make it feasible to complete AutoML tuning. That many of the algorithms are not fully parallizable makes it even more important.

When sampling a dataset I assume there are two key criteria: representability and versatility. The partial sample should represent the dataset as a whole, but it should also include edge cases.

At the moment, I am taking random subsample using SplitTrainTestSet method. I assume it is representative of the whole dataset by virtue of randomness.

However, I would like ways make sure I am not removing edge cases by accident.

Feature requests:
- Method to remove duplicates from the dataset
- Method to remove near duplicates (inspired by this article https://huggingface.co/blog/dedup )

Question:
- Are there ways or methods to analyze subsamples representativeness? Perhaps plotting, or auto-analyzing, the distribution of values in each column. However, this only looks at the columns individually.
- Methods to analyze edge cases not being dropped?

I expect not everything can and should be supported by Microsoft.ML so I will appreciate any ideas and insights on how to ideally complete these tasks within .NET ecosystem. I think data deduplication maybe might be a good addition for Microsoft.ML, and if so, why not near deduplication.