should we reduce the dimensionality of topic_model.topic_embeddings_ ?

Nobody has claimed this yet.

Dear Creator of the amazing BERTopic

I want to perform cosine similarity of the topic_embeddings to a list of labels. I found it to perform better than zeroshot (and faster !) for my use case. However, the embeddings in topic_model.topic_embeddings_ are 384 dimensional vectors, ie their dimension is not reduced using hdbscan. To my understanding, the cosine similarity could suffer from the curse of dimensionality because of that. Actually, ploting the max cosine similarity to my list of labels might suggest that, making most topics cosine similar to my labels to 0. 55 :

Should I add a dimensionality reduction step ? would it be possible to use directly the reduced embeddings for the topic_model ?

Again thanks for your work

Start by tracing how BERTopic produces and stores topic_embeddings_, then compare that path with the HDBSCAN dimensionality-reduction step mentioned in the issue. Determine whether reduced embeddings can be used for cosine similarity without changing topic behavior; completion would require a clear design decision and validation of the resulting similarity scores.

Written by the indexing model from the issue text.