cleanlab / cleanlab/cleanlab

DBSCAN/HDSBCAN modify distance matrix in-place

Open
#876 0 comments 0 reactions 0 assignees View on GitHub
enhancement help-wanted
Dominant language
Python
Stars
11.7k
Forks
920
PR merge metrics
No merged PRs in 30d

Description

Both DBSCAN and HDBSCAN modify the distance matrix passed to the `fit` method. This modification affects the [underperforming group](https://github.com/cleanlab/cleanlab/pull/838) issue manager (under development at the time of creating this issue).

* Modification of the distance matrix can cause unexpected results from Datalab. Since the computed KNN graph is stored as part of `info`, any issue managers that reuse the knn-graph generated by the underperforming group issue manager might not function as expected. Further, computation of `nearest_neighbor` and `distance_to_nearest_neighbor` as part of the issue manager's info can also produce incorrect output.
* A temporary solution for this problem involves passing a copy of the distance matrix to the `fit` method. However, creating a copy of a large distance matrix can be memory intensive for users.

This behaviour is being tracked in Scikit-Learn's repository [[27508](https://github.com/scikit-learn/scikit-learn/issues/27508)]. However, the proposed solution for Scikit-Learn's future releases is to internally create a copy of the distance matrix, which will not fix the memory issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.