MaartenGr / MaartenGr/BERTopic

`precomputed` Distance Compatibility for HDBSCAN

Open
#1,879 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hi there!

Recently, I've been experimenting with the UMAP + HDBSCAN workflow and noticed an opportunity to enhance its functionality related to distance metrics.

#### Proposal:
I propose to add compatibility for `precomputed` distances in HDBSCAN within BERTopic. This would allow users to use custom distance metrics, including the **cosine** similarity, which is not directly supported as a built-in metric in HDBSCAN.

#### Why This Matters:

- Flexibility: This addition would provide users with the ability to use a broader range of distance metrics, tailoring the model more closely to their specific needs.
- Semantic Understanding: Cosine similarity is particularly effective for understanding semantic relationships in text data. By enabling precomputed distances, users can leverage cosine similarity for better topic modeling outcomes.
- Wider Application: This feature could broaden BERTopic's applicability across different domains where specific distance metrics are crucial for accurate modeling.

#### Implementation Insight:
I've already implemented (very quick) this feature locally and found that it integrates well with the existing pipeline. I'm confident that it could be a valuable addition to BERTopic without compromising performance or usability. The following is an non-exhaustive way of implementing this, of course this will need more work to be fully incorporated, but is just a mock of it:
```python
def __init__(self,
language: str = "english",
top_n_words: int = 10,
n_gram_range: Tuple[int, int] = (1, 1),
min_topic_size: int = 10,
nr_topics: Union[int, str] = None,
low_memory: bool = False,
calculate_probabilities: bool = False,
seed_topic_list: List[List[str]] = None,
zeroshot_topic_list: List[str] = None,
zeroshot_min_similarity: float = .7,
embedding_model=None,
umap_model: UMAP = None,
hdbscan_model: hdbscan.HDBSCAN = None,
vectorizer_model: CountVectorizer = None,
ctfidf_model: TfidfTransformer = None,
representation_model: BaseRepresentation = None,
verbose: bool = False,
distance_matrix: np.ndarray = None, <--------------------
):
```
```python
self.hdbscan_model = hdbscan_model or hdbscan.HDBSCAN(min_cluster_size=self.min_topic_size,
metric='euclidean',
cluster_selection_method='eom',
prediction_data=True)
self.distance_matrix = distance_matrix <--------------------

```
```python
def _cluster_embeddings(self,
umap_embeddings: np.ndarray,
documents: pd.DataFrame,
partial_fit: bool = False,
y: np.ndarray = None) -> Tuple[pd.DataFrame,
np.ndarray]:
...
logger.info("Cluster - Start clustering the reduced embeddings")
if partial_fit:
self.hdbscan_model = self.hdbscan_model.partial_fit(umap_embeddings)
labels = self.hdbscan_model.labels_
documents['Topic'] = labels
self.topics_ = labels
elif self.hdbscan_model.get_params()["metric"] == "precomputed": <--------------------
logger.info("Cluster - Using a precomputed distance matrix (MUST BE OF THE REDUCED EMBEDDINGS)")
self.hdbscan_model.fit(self.distance_matrix)
labels = self.hdbscan_model.labels_
documents['Topic'] = labels
self._update_topic_size(documents)
```

I'd love to hear your thoughts on this proposal. Do you see this as a valuable addition to BERTopic? Would there be any concerns or additional considerations we should account for?

I'm excited about the potential to contribute this feature to the community and look forward to your feedback.

Thank you for considering this enhancement!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the BERTopic constructor and `_cluster_embeddings`, focusing on how the provided `hdbscan_model` is configured and how clustering updates document topics. Check the proposed precomputed distance matrix flow and determine the validation and compatibility requirements; done means the feature works for reduced-embedding distances without breaking the existing pipeline.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.