Compare LDA, NMF, LSA with BERTopic (w/ embedding: all-MiniLM-L6-v2 + dim_red: UMAP + cluster: HDBSCAN)

Nobody has claimed this yet.

Hi @MaartenGr ,

Given a dataset of texts, we want to extract topics using LDA, NMF, LSA and BERTopic (w/ embedding: all-MiniLM-L6-v2 + dim_red: UMAP + cluster: HDBSCAN).

In order to select the best algorithm for this dataset, there was an intuition that a mathematical combination of an applicable topic coherence measure and an applicable topic diversity measure was chosen to optimize. In one of previous issues, [#90](https://github.com/MaartenGr/BERTopic/issues/90) , I observed that when calculating topic coherence, you treated concatenation of texts belonging to a cluster as a single document.

However, for calculating topic coherence for LDA, LSA and NMF, we simply get the BoW representation of given texts and calculate topic coherence.

To the best of my understanding, shouldn't we ensure that the corpus and dictionary passed to initialize CoherenceModel object from gensim.coherencemodel be the same between BERTopic and LSA/LDA/NMF, so that we can actually now compare values of topic coherence achieved for all algorithms and then select the one with highest topic coherence?

Apologies for such a long description.

Thanks,
Abi

Start with issue #90 and the gensim.coherencemodel.CoherenceModel setup described here. Compare how BERTopic and the LDA, NMF, and LSA approaches construct the corpus and dictionary, including the all-MiniLM-L6-v2, UMAP, and HDBSCAN configuration. Done should be a reproducible, agreed method for comparing topic coherence and diversity across the algorithms.

Written by the indexing model from the issue text.