MaartenGr / MaartenGr/BERTopic
How can i use "precomputed" HDBSCAN in BerTopic?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
BerTopic Version = 0.16.4
```
umap_model = umap.UMAP(n_neighbors=15, n_components=24, min_dist=0.0, metric='cosine', random_state=100)
embedding_model = SentenceTransformer('sentence-transformers/paraphrase-multilingual-mpnet-base-v2')
hdbscan_model = HDBSCAN(metric='euclidean',min_cluster_size=10, cluster_selection_method='eom', prediction_data=True)
model = BERTopic( hdbscan_model=hdbscan_model,embedding_model=embedding_model,umap_model=umap_model, language="english", calculate_probabilities=True)
topics, probabilities =model.fit_transform(sentecnes)
```
as you know when i run the above code there is no problem and every thing is ok , but i must use the hdbscan as below:
```
hdbscan_model = HDBSCAN(metric='precomputed',min_cluster_size=10, cluster_selection_method='eom', prediction_data=True)
cluster_labels = hdbscan_model.fit_predict(distances)
```
because i have a pairwaise matrix "distances[ ]" as the distances between my embeddings.
with this metric='precomputed' i cant't run my bertopic model and the error is :
ValueError: operands could not be broadcast together with shapes (24,) (24,489)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the BERTopic 0.16.4 example with the shown UMAP, SentenceTransformer, HDBSCAN, distances matrix, and model.fit_transform call. Compare the working Euclidean configuration with metric='precomputed' and the reported broadcasting error; done means the supported behavior or limitation is established and documented with a reproducible result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100