MaartenGr / MaartenGr/BERTopic

Scikit-learn's HDBSCAN Implementation

Open
#2,031 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement question
Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

In a recent version of scikit-learn, I believe it was [v1.3](https://scikit-learn.org/1.5/whats_new/v1.3.html#id8), HDBSCAN was implemented with base functionality. Considering scikit-learn is already a requirement of BERTopic it stands to reason to use that implementation instead of the [original implementation](https://github.com/scikit-learn-contrib/hdbscan) since scikit-learn has more contributors. Moreover, common installation issues related to HDBSCAN might be alleviated with this.

There are a couple of issues worth mentioning:
* Calculation of probabilities is if I'm not mistaken, not implemented in scikit-learn's HDBSCAN
* A solution would be to use the cosine similarities as the default method of calculating probabilities
* The feature set is smaller than the original implementation
* Speed needs to be tested to identify whether this is worth it
* Accuracy, whatever that means in this context, might also need some exploration

For those reading this, I'm interested to hear what you all think about this suggested change!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing scikit-learn's HDBSCAN with the original HDBSCAN implementation currently used by BERTopic. Evaluate probability calculation, feature coverage, speed, and accuracy; done means the trade-offs are documented and there is a clear decision on whether replacement is viable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.