Topic embedding, top n representative documents and topic distribution

Nobody has claimed this yet.

Hi @MaartenGr,

Thanks for the great package and all the documentations. I really learn a lot!
Just have several clarified questions.
1. topic_embeddings_

From the document, `topic_embeddings` has been calculated as follows:

> topic_embeddings_: The embeddings for each topic. It is calculated by taking the weighted average of word embeddings in a topic based on their c-TF-IDF values.

But I also saw a new issue where you mentioned `topic_embeddings` is the centroid of a cluster if we are using embedding_model. Whether the centroid of a cluster here is calculated by averaging all the sentence embeddings in a cluster?

3. Find top n representative documents:

From #848, you mentioned that

> In the v0.14 release of BERTopic, all representative documents are extracted in the same way regardless of whether you are using HDBSCAN or another clustering algorithm. A random subset of 500 documents is sampled for each cluster after which we use c-TF-IDF to score those documents. The resulting values are compared with their topic's c-TF-IDF values to rank the documents based on their closeness to a topic.

Since I am using v0.15, this means that the top 50 representative documents I get for each topic is extracted by comparing the topic's c-TF-IDF value to each document's c-TF-IDF value instead of comparing the topic embedding (avg of sentences embeddings) to each sentence's embedding. Is that correct?

5. Topic distribution of new/existing documents

I am also curious about the topic distribution of the existing documents. I noticed that there are two ways to visualize the topic distribution using `.approximate_distribution()`. First is comparing the c-TF-IDF calculations between the token sets and all topics. Second is selecting embedding_model instead to do this comparison. Do this mean to compare the topic_embedding(avg of sentences embeddings within this topic) with the token set embedding?

Also, when I check the topic distribution of say sentence 1, its closest topic is different from the assigned topic by HDBSCAN with c-TF-IDF method. For example, its closest topic is Topic2 but its assigned topic is Topic3. Whether this is also the case for embedding method? Additionally, the cosine similarity between the topic and sentence embedding with embedding method is very low (mostly around 0.006). Whether this will be an issue?

Thanks for your help!! 

Start with the BERTopic documentation and the APIs named in the issue: topic_embeddings_, representative documents, and approximate_distribution(). Trace the c-TF-IDF and embedding paths, then document how each is calculated, why assignments can differ, and how to interpret low cosine similarities; done means each numbered question has a definitive, version-specific answer.

Written by the indexing model from the issue text.