BERTopic on big data and probabilities

Nobody has claimed this yet.

Hello,
I've been following the instructions provided in the guide - https://colab.research.google.com/drive/1W7aEdDPxC29jP99GGZphUlqjMFFVKtBC?usp=sharing#scrollTo=oGQnu6lO9SrE
particularly the section concerning the manual implementation of BERTopic. Upon execution, I specified calculate_probabilities=True for BERTopic. However, after the model fitting with the code snippet below, the expected probabilities aren’t returned.

Could you please provide insight or a solution to modify the code so that it successfully returns the probability matrix as part of the output from the hdbscan_model?

Thank you for your assistance.

````
from cuml.cluster import HDBSCAN
from sentence_transformers import SentenceTransformer

from bertopic import BERTopic
from bertopic.cluster import BaseCluster
from bertopic.representation import KeyBERTInspired

class Dimensionality:
""" Use this for pre-calculated reduced embeddings """
def __init__(self, reduced_embeddings):
self.reduced_embeddings = reduced_embeddings

def fit(self, X):
return self

def transform(self, X):
return self.reduced_embeddings

# Prepare sub-models
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
umap_model = Dimensionality(reduced_embeddings)
hdbscan_model = BaseCluster()
vectorizer_model = CountVectorizer(vocabulary=vocab, stop_words="english")
representation_model = KeyBERTInspired()

# Fit BERTopic without actually performing any clustering
topic_model= BERTopic(
embedding_model=embedding_model,
umap_model=umap_model,
hdbscan_model=hdbscan_model,
vectorizer_model=vectorizer_model,
representation_model=representation_model,
calculate_probabilities= True,
verbose=True
).fit(texts, embeddings=embeddings, y=clusters)
```

Start with the custom Dimensionality class, the BaseCluster assignment, and the BERTopic.fit call in the reported snippet. Check how calculate_probabilities is handled when clustering is bypassed, then verify whether fitting with the supplied embeddings and clusters produces a probability matrix as expected.

Written by the indexing model from the issue text.