MaartenGr / MaartenGr/BERTopic
Scikit-learn's HDBSCAN Implementation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
In a recent version of scikit-learn, I believe it was [v1.3](https://scikit-learn.org/1.5/whats_new/v1.3.html#id8), HDBSCAN was implemented with base functionality. Considering scikit-learn is already a requirement of BERTopic it stands to reason to use that implementation instead of the [original implementation](https://github.com/scikit-learn-contrib/hdbscan) since scikit-learn has more contributors. Moreover, common installation issues related to HDBSCAN might be alleviated with this.
There are a couple of issues worth mentioning:
* Calculation of probabilities is if I'm not mistaken, not implemented in scikit-learn's HDBSCAN
* A solution would be to use the cosine similarities as the default method of calculating probabilities
* The feature set is smaller than the original implementation
* Speed needs to be tested to identify whether this is worth it
* Accuracy, whatever that means in this context, might also need some exploration
For those reading this, I'm interested to hear what you all think about this suggested change!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing scikit-learn's HDBSCAN with the original HDBSCAN implementation currently used by BERTopic. Evaluate probability calculation, feature coverage, speed, and accuracy; done means the trade-offs are documented and there is a clear decision on whether replacement is viable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100